TestPrism: Rethinking Test Evaluation Beyond a Single Reference

Key points

• However, the common practice of evaluating tests against a single reference solution overlooks alternative valid implementations and can overstate test quality.1
• We introduce TestPrism, comprising 300 test tasks from 17 sources and 3000 candidate implementations, evenly split between valid and invalid solutions.1
• To address these weaknesses, we introduce TestHelix, which combines heterogeneous synthesis of test and repair pairs with peer cross validation and recursive self improvement (RSI).1
• Across two models, TestHelix improves Joint Success Function by 8.67 to 9.00 percentage points over the native harness comparators in the TestHelix evaluation1

Sources (1)

• [1]TestPrism: Rethinking Test Evaluation Beyond a Single Reference
  Hugging Face Daily Papers · Oct 8, 12:00 AM

  Large language model (LLM) coding agents have advanced test generation across diverse programming tasks.

  However, the common practice of evaluating tests against a single reference solution overlooks alternative valid implementations and can overstate test quality.

Extractive summary: sentences quoted from the sources.