Efficient Best-of-N policy evaluation for inference-time alignment

Key points

• Evaluating BoN policies from logged data is challenging under sample-only access because standard off-policy estimators require density ratios that depend on unavailable response likelihoods.1
• In this paper, we propose a sample-only framework for evaluating and selecting BoN policies without access to these likelihoods.1
• We show that the order-statistic structure of BoN allows the required density ratios to be expressed through score-rank probabilities that are estimable from samples alone.1
• We establish valid asymptotic inference even under reward estimator misspecification and prove the efficiency of our BoN-DR estimator.1

Sources (1)

• [1]Efficient Best-of-N policy evaluation for inference-time alignment
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 12:26 AM

  Best-of-N (BoN) is a common inference-time alignment method that selects the highest-scoring response among N samples from a reference model.

  Evaluating BoN policies from logged data is challenging under sample-only access because standard off-policy estimators require density ratios that depend on unavailable response likelihoods.

Extractive summary: sentences quoted from the sources.