Mind the Spatial Gap: A Benchmark for Geospatial Reasoning and Constrained Spatial Editing

Key points

• Understanding and manipulating spatial relations is fundamental to AI systems that reason about physical environments and construct grounded world models.1
• However, existing evaluations of multimodal large language models (MLLMs) for spatial reasoning rely predominantly on static question-answering (QA) over small-scale or synthetic scenes, rarely probing whether models can modify spatial configurations while satisfying explicit geometric constraints.1
• Grounded in real-world urban data across six geographically diverse cities, GeoSpatialBench evaluates a comprehensive taxonomy of 28 topological, directional, metric, and ordering relations across structured GeoJSON and map image modalities, with ground-truth references and constraints verified via GIS operations.1
• Our empirical evaluation across frontier MLLMs uncovers a stark spatial gap: structured GeoJSON consistently outperforms map images, with multimodal combinations offering no reliable gain over GeoJSON alone, and while models show reasonable competence on qualitative topological and ordering, performance drops on quantitative metric relations.1

Sources (1)

• [1]Mind the Spatial Gap: A Benchmark for Geospatial Reasoning and Constrained Spatial Editing
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 02:16 AM

  We introduce GeoSpatialBench, a benchmark designed to evaluate an escalating progression of spatial cognition, ranging from spatial relation understanding and geometric reasoning to constrained spatial editing.

  Understanding and manipulating spatial relations is fundamental to AI systems that reason about physical environments and construct grounded world models.

Extractive summary: sentences quoted from the sources.