Agreement Without Alignment: Label-Free Rubric Clarification Increases LLM-Judge Consensus but Not Agreement with Humans
LLM judges are increasingly deployed as panels, and disagreement among panel members is often read as a symptom of an unclear rubric.
ProofPaper ↗
Key points
- We test a label-free procedure that turns this disagreement into rubric repairs.
- An additional stage that moves text-checkable sub-criteria to deterministic checkers transferred none in our evaluation.
- We pre-specified four hypotheses in a version-controlled protocol and evaluated six rubric conditions with a panel of five judges from three providers on SummEval, the HealthBench meta-evaluation, and USR Topical-Chat, for a total of 129,770 judge requests, excluding retries.
- Selecting clauses by disagreement makes judges agree with each other more than editing arbitrary clauses does, but in our setting the added consensus did not bring the judges closer to human annotators.
Sources (1)
- [1]Agreement Without Alignment: Label-Free Rubric Clarification Increases LLM-Judge Consensus but Not Agreement with HumansarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 09:49 AM
LLM judges are increasingly deployed as panels, and disagreement among panel members is often read as a symptom of an unclear rubric.
We test a label-free procedure that turns this disagreement into rubric repairs.
Extractive summary: sentences quoted from the sources.