Certified by Abstention: Distribution-Free Guarantees for Chain-of-Thought Verifiers at Small Calibration Budgets

Key points

• Signals that predict whether a chain-of-thought (CoT) trace is correct are compared by AUC, but deploying one requires a threshold with a guarantee.1
• The central observation is validity by abstention: an $(α,δ)$-valid procedure that issues a certificate with probability $P{\rm fire}$ bounds the failure probability of an issued certificate only by $δ/P{\rm fire}$, so a certificate that rarely fires can be valid and wrong every time it is used.1
• We then give a floor-started fixed-sequence certificate, valid without monotonicity assumptions, that covers more than the Bonferroni certificate on every model-signal pair and raises coverage at the non-vacuous target $0.75π0$ from 0.05 to 0.16, although the floor keeps absolute coverage small.1
• Finally, a certificate cannot see what matters after deployment: under benchmark shift the error among accepted traces tracks the new task's base error, and under best-of-$n$ selection against the verifier it rises past the target while the empirical failure frequency stays below $δ$, because abstention absorbs the failures.1

Sources (1)

• [1]Certified by Abstention: Distribution-Free Guarantees for Chain-of-Thought Verifiers at Small Calibration Budgets
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 06:40 AM

  We ask what distribution-free selective guarantees deliver for CoT verifiers at realistic calibration budgets of tens to a few hundred labelled problems, using seven open models, five verifier signals and 37,000 graded traces.

  Signals that predict whether a chain-of-thought (CoT) trace is correct are compared by AUC, but deploying one requires a threshold with a guarantee.

Extractive summary: sentences quoted from the sources.