World Models' Last Exam in Physics

Key points

• Video world models can produce visually convincing yet physically inconsistent sequences, raising concerns about their reliability for prediction and planning in embodied AI systems.1
• Existing evaluations often rely on model-based judgments or reference videos, while direct physical tests largely focus on mechanics.1
• Experiments on eight video generation models across 1,280 videos reveal persistent physical inconsistencies and substantial variation across tasks, with the best model achieving an overall score of 57.76 out of 100.1
• Evaluation on synthetic videos with known physical relationships provides evidence for the validity of the measurement module under controlled conditions.1

Sources (1)

• [1]World Models' Last Exam in Physics
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 05:59 PM

  We introduce World Models' Last Exam in Physics, a measurement-based benchmark for evaluating physical consistency in video world models.

  Video world models can produce visually convincing yet physically inconsistent sequences, raising concerns about their reliability for prediction and planning in embodied AI systems.

Extractive summary: sentences quoted from the sources.