Domain Adaptation of SAM 2.1 for Building-Component Segmentation in Image-Based 3D Reconstruction

Key points

• In order to benefit from consistent grouping of the 2D masks in 3D environments, vision foundation models such as the Segment Anything Model have been utilized by Group Anything with Radiance Fields (GARField) to perform 3D segmentation.1
• This study fine-tuned SAM 2.1 on the HBD dataset, consisting of 2,679 annotated images from 42 buildings, to enhance multi-class segmentation of 2D building images, covering 11 classes (Wall, Window, Ceiling, Door, Column, Beam, Floor, Opening, Curtain Wall, Roof, and Lift).1
• The masks in the building images were grouped in the neural radiance field, and the pretrained SAM 2.1 was compared with the fine-tuned SAM 2.1 in the newly rendered images.1
• This advances the application of Group Anything with Radiance Fields (GARField) in 3D, enabling better segmentation of building structures for 3D auto-reconstruction.1

Sources (1)

• [1]Domain Adaptation of SAM 2.1 for Building-Component Segmentation in Image-Based 3D Reconstruction
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 10:43 PM

  3D reconstruction of building components using image-based scanning requires appropriate segmentation of the objects in the 2D building images.

  In order to benefit from consistent grouping of the 2D masks in 3D environments, vision foundation models such as the Segment Anything Model have been utilized by Group Anything with Radiance Fields (GARField) to perform 3D segmentation.

Extractive summary: sentences quoted from the sources.