Align Before You Combine: Reference Space Calibration for Supervision Without Ground Truth

Key points

• Our framework aligns subset-specific scorers using a synthetic ordinal reference space before fusion.1
• This reference space is constructed from ordered calibration features that represent the latent concept, providing a common scale on which otherwise incomparable scorer outputs can be aligned.1
• Across three benchmark datasets, our framework consistently outperforms uncalibrated averaging and achieves higher primary-metric point estimates on the evaluation metrics than the best individual scorer.1
• Additionally, we show that using fewer calibration levels per feature can closely approximate higher-resolution results at substantially lower computational cost.1

Sources (1)

• [1]Align Before You Combine: Reference Space Calibration for Supervision Without Ground Truth
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 06:18 AM

  We introduce a calibration-first framework that produces supervision scores without access to ground-truth labels or a shared annotation space.

  Our framework aligns subset-specific scorers using a synthetic ordinal reference space before fusion.

Extractive summary: sentences quoted from the sources.