Do Vision Models Learn Physical Constraints or Rendering Shortcuts? A Counterfactual Benchmark for Grounded Physical Consistency

Key points

• Modern image editing models can satisfy a text instruction while breaking the physics of the edited scene.1
• A new object may cast no shadow, a mirror may fail to reflect visible geometry, or an object may float above a surface that should support it.1
• We introduce a counterfactual benchmark whose controlled synthetic component uses Mitsuba 3 to generate 5,500 images from 500 scene families.1
• The renderer pipeline provides category labels, affected-region masks and boxes, scene metadata, and explanation targets.1

Sources (1)

• [1]Do Vision Models Learn Physical Constraints or Rendering Shortcuts? A Counterfactual Benchmark for Grounded Physical Consistency
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 11:06 PM

  We study physical plausibility diagnosis, detecting whether an edited image violates scene physics, naming the violation type, localizing the affected region, and explaining the failure in language.

  Modern image editing models can satisfy a text instruction while breaking the physics of the edited scene.

Extractive summary: sentences quoted from the sources.