GeoFitBench-LLM: A Benchmark for Assessing Large Language Models' Capabilities in Geospatial Data Requirements Elicitation for Geo-Analytic Tasks

Key points

• Large language models are becoming central to GeoAI systems, yet their evaluation rarely considers the elicitation of geospatial data requirements.1
• This omission is critical as retrieval oriented geo-analytic tasks cannot be supported reliably without first identifying the necessary datasets and constraints.1
• The benchmark is structured around two reasoning dimensions, Spatio-Temporal Relational and Semantic Grounding, reflecting long-standing challenges in geospatial information retrieval and related natural language tasks.1
• Ground truth requirements are encoded in a fixed, provider-independent schema for reproducible evaluation of resource identification and attribute-level specification.1

Sources (1)

• [1]GeoFitBench-LLM: A Benchmark for Assessing Large Language Models' Capabilities in Geospatial Data Requirements Elicitation for Geo-Analytic Tasks
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 07:45 PM

  GeoFitBench-LLM is introduced as a benchmark for assessing whether LLMs can infer the datasets, the supporting resources and the constraints required by geo-analytic requests in natural language.

  Large language models are becoming central to GeoAI systems, yet their evaluation rarely considers the elicitation of geospatial data requirements.

Extractive summary: sentences quoted from the sources.