Pumpire: Unified Benchmark for Metric Distance Estimation

Key points

• In contrast to previous approaches that normally evaluate depth and camera intrinsics separately or evaluate point-clouds with geometric similarity metrics, which cannot directly reflect models' point-to-point distance estimation capability, Pumpire directly assesses point-to-point distances from the reconstructed geometry.1
• To this end, we collect a large-scale and diverse dataset (pumpire-6k) comprising 100 real-world scenes, each annotated with physically measured point-pair distances and containing 64 frames, for a total of 6,400 frames.1
• Building on this dataset, we establish a holistic evaluation protocol that covers both image- and video-level 3D foundation models and enables direct assessment of point-pair distance errors and cross-setting comparison.1
• By offering this benchmark, we target the more fundamental ability to perceive and estimate physical scale in the reconstructed 3D space, which prior evaluation protocols have largely overlooked.1

Sources (2)

• [1]Pumpire: Unified Benchmark for Metric Distance Estimation
  Hugging Face Daily Papers · Oct 8, 12:00 AM

  We present Pumpire, a unified benchmark for evaluating metric point-pair distance estimation capability of both image- and video-level 3D foundation models, with or without depth priors.

  In contrast to previous approaches that normally evaluate depth and camera intrinsics separately or evaluate point-clouds with geometric similarity metrics, which cannot directly reflect models' point-to-point distance estimation capability, Pumpire directly assesses point-to-point distances from the reconstructed geometry.

• [2]Pumpire: Unified Benchmark for Metric Distance Estimation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 05:55 PM · same content

Extractive summary: sentences quoted from the sources.