ROSETTA: Rotation Subspace Estimation for Training-free Transmodal Alignment

Key points

• When a query must be matched against a database built with a different sensor or processed by a different encoder, the independently trained encoders impose distinct coordinate systems on the feature space.1
• However, the cross-covariance between encoder outputs has low effective rank, constraining only a small fraction of the rotation parameters, therefore local alignment is tractable.1
• ROSETTA is a training-free calibration layer that estimates the local rotation via an expectation-maximization procedure, requiring no coobservation at query time.1
• We validate ROSETTA on three public driving datasets spanning LiDAR, camera, thermal, and event modalities in cross-modal, cross-device, cross-representation, and cross-backbone settings, with milliseconds overhead per query.1

Sources (1)

• [1]ROSETTA: Rotation Subspace Estimation for Training-free Transmodal Alignment
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 01:57 AM

  We show that the relative transformation between any two such systems is a spatially preserving rotation in whitened feature space, whose angular velocity exhibits white-noise characteristics: large in norm, temporally uncorrelated, and unpredictable from the features themselves.

  When a query must be matched against a database built with a different sensor or processed by a different encoder, the independently trained encoders impose distinct coordinate systems on the feature space.

Extractive summary: sentences quoted from the sources.