From Structure to Validity: Mechanism-Guided Generation and Refinement of LLM Rubrics
Guided by these findings, we introduce Adaptive Rubric Tree Search with Hybrid refinement (ARTS-Hybrid), which combines adaptive tree construction with semantic-first, response-calibrated refinement.
ProofPaper ↗
Key points
- Rubrics have been widely used to evaluate and optimize language models on open-ended tasks.
- In this work, we conduct a systematic mechanistic analysis of rubric design across criterion organization, criterion operationalization, and score aggregation.
- Our analysis shows that structural organization enables more comprehensive criterion construction, semantic specification alone does not ensure effective criterion-level judgments, and sophisticated score aggregation provides limited additional gains.
- Experiments across benchmarks show that ARTS-Hybrid improves both end-to-end evaluation performance and intrinsic rubric quality.
Sources (1)
- [1]From Structure to Validity: Mechanism-Guided Generation and Refinement of LLM RubricsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 10:48 AM
Guided by these findings, we introduce Adaptive Rubric Tree Search with Hybrid refinement (ARTS-Hybrid), which combines adaptive tree construction with semantic-first, response-calibrated refinement.
Rubrics have been widely used to evaluate and optimize language models on open-ended tasks.
Extractive summary: sentences quoted from the sources.