SpatialASM: Internalizing Active Mentalization for Spatial Reasoning

Key points

• When solving complex spatial reasoning tasks, ordinary humans naturally revisit the scenes from task-relevant viewpoints and mentally reconstruct the spatial layouts from those viewpoints.1
• But can Large Vision-Language Models (LVLMs) proactively think in the same way?1
• SpatialASM employs an allocentric novel-view generator to ground hypothetical viewpoints in visual evidence and produces two complementary forms of supervision: i) explicit ASM trajectories that teach LVLMs when and how to adaptively invoke view planning and imagination during spatial reasoning, and ii) automatically-constructed perspective-taking tasks that further strengthens viewpoint-conditioned scene imagination.1
• On seven benchmarks spanning relatively perception-oriented spatial understanding to more challenging tasks requiring multi-image, cross-view and viewpoint-transformation reasoning, SpatialASM yields performance gain on LVLMs of four LVLM backbones in spatial reasoning while retaining their original general vision capabilities.1

Sources (2)

• [1]SpatialASM: Internalizing Active Mentalization for Spatial Reasoning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 04:39 PM

  We term this ability Active Spatial Mentalization (ASM), an intrinsic component of visual intelligence that remains substantially underdeveloped in current LVLMs. To bridge this gap, we introduce SpatialASM, a scalable framework to internalize ASM into LVLMs through training-time spatial simulation.

  When solving complex spatial reasoning tasks, ordinary humans naturally revisit the scenes from task-relevant viewpoints and mentally reconstruct the spatial layouts from those viewpoints.

• [2]SpatialASM: Internalizing Active Mentalization for Spatial Reasoning
  Hugging Face Daily Papers · Oct 9, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.