GeopoliticalQA: Benchmarking Retrieval and Generation over Intelligence Reports

Key points

• Geopolitical and strategic intelligence reports constitute a rich source of expert knowledge and have increasing value in the current political climate, but their heterogeneity and analytical complexity pose challenges for retrieval-augmented question answering.1
• We propose a document-aware generation pipeline that profiles reports before generating questions and validate dataset quality through expert annotation and an LLM-as-judge protocol.1
• Using the benchmark, we evaluate 15 retrieval systems and six open-weight LLMs for RAG.1
• We find that models perform similarly on document-level retrieval, while at the harder chunk level retrieval quality is not driven by embedding-model scale: bge-m3, a mid-size open encoder matches an order of magnitude larger ones and outperforms commercial embedding APIs.1

Sources (1)

• [1]GeopoliticalQA: Benchmarking Retrieval and Generation over Intelligence Reports
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 06:35 PM

  We introduce a benchmark for grounded question answering over 415 open-source geopolitical and strategic intelligence reports, comprising 5,525 document-grounded question-answer pairs spanning factual, inferential, comparative, and risk-oriented questions.

  Geopolitical and strategic intelligence reports constitute a rich source of expert knowledge and have increasing value in the current political climate, but their heterogeneity and analytical complexity pose challenges for retrieval-augmented question answering.

Extractive summary: sentences quoted from the sources.