Hardware-aware Calibrated Clustered Attention for Efficient Visual Geometric Transformers

Key points

• The Visual Geometry Grounded Transformer (VGGT) marks a significant leap forward in 3D scene reconstruction, as it is the first model that directly infers all key 3D attributes (camera poses, depths, and dense geometry) jointly in one pass.1
• By limiting the clustering within HW-friendly neighborhood blocks, BC attention reduces the computation overhead of query clustering as well as the costly data movement between on- and off-chip memory.1
• Moreover, we introduce a hashing hyperplane calibration method and a threshold-based error compensation method to reduce clustering errors efficiently, which is a bottleneck in the current clustered attention mechanism.1
• Overall, our experiments on GPU demonstrate that calibrated BC attention accelerates the global attention layers by 2.10-2.63$\times$ and the whole backbone by 1.77-2.35$\times$ with negligible loss (1%) for large scenes.1

Sources (1)

• [1]Hardware-aware Calibrated Clustered Attention for Efficient Visual Geometric Transformers
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 01:13 AM

  In this paper, we propose blockwise clustered attention (BC attention) to accelerate the global attention layers in VGGT.

  The Visual Geometry Grounded Transformer (VGGT) marks a significant leap forward in 3D scene reconstruction, as it is the first model that directly infers all key 3D attributes (camera poses, depths, and dense geometry) jointly in one pass.

Extractive summary: sentences quoted from the sources.