MARGIN: Runtime Confidence Calibration for Multi-Agent Foundation Model Coordination

Key points

• When a coordinator compares answers from heterogeneous foundation models, self-reported confidence may have different meanings across responders and changing workloads.1
• MARGIN tracks recent accuracy and stated confidence within confidence bands, uses their ratio to correct reported confidence, and blends sparse-band corrections toward a model-level estimate.1
• Against five online calibration baselines receiving identical feedback and retaining their learned state across each transition, MARGIN achieves lower post-shift expected calibration error than all five in two code-generation transitions and than four in a question-answering transition; the remaining question-answering comparison is inconclusive.1
• In separate code-generation coordination experiments, calibration improves the ranking of correct responses and increases answer-selection accuracy by 4.3 and 14.0 percentage points on two of three benchmarks relative to uncalibrated confidence weighting.1

Sources (1)

• [1]MARGIN: Runtime Confidence Calibration for Multi-Agent Foundation Model Coordination
  Hugging Face Daily Papers · Oct 8, 12:00 AM

  This paper presents MARGIN (Multi-Agent Runtime Grading via Incremental Normalisation), a runtime calibration method that learns model-specific confidence corrections from observed answer outcomes without retraining the models or requiring a held-out calibration set.

  When a coordinator compares answers from heterogeneous foundation models, self-reported confidence may have different meanings across responders and changing workloads.

Extractive summary: sentences quoted from the sources.