Reference-Free Food Volume Estimation from Egocentric Stereo Video

Key points

• Current stereo vision-based approaches for food portion estimation are hindered by the user burden of using a mobile phone to capture multiple images or placing a reference object in the scene for scale calibration.1
• By utilizing FoundationStereo for zero-shot stereo matching, SAM 2 for food segmentation, and TSDF fusion over a set of keyframes, our method yields a visually and metrically accurate 3D mesh of any plated food item from a single stereo recording.1
• We leverage the unique offerings of wearables, such as a factory-calibrated stereo baseline and camera poses estimated on-device via visual-inertial odometry (VIO), to perform fully automatic reconstruction without a scale reference object in the scene.1
• We show strong initial volume estimation results across five recorded food items, achieving a mean absolute percentage error of 9.1%.1

Sources (1)

• [1]Reference-Free Food Volume Estimation from Egocentric Stereo Video
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 09:30 PM

  As a step towards passive dietary intake monitoring, we introduce a method that uses stereo video from a wearable egocentric device to reconstruct food items in true physical scale to accurately estimate volume.

  Current stereo vision-based approaches for food portion estimation are hindered by the user burden of using a mobile phone to capture multiple images or placing a reference object in the scene for scale calibration.

Extractive summary: sentences quoted from the sources.