Evaluating Exact Output and Checkpoint-State Prediction in Real Programs

Key points

• It extends CRUXEval-style output prediction with paired shorter- and longer-trace inputs and checkpoints inside and after a loop.1
• The benchmark contains 400 cases from 371 Python and C++ programs, evaluated under seven settings from four model families without tools or code execution.1
• The strongest setting scores 93.0% on shorter-trace final output, 77.0% on longer-trace final output, and 65.5% and 63.5% on the two state tasks; these scores also hold when missing responses count as wrong.1
• Across 2,397 matched Python comparisons with identical source, changing to the longer-trace input yields 528 correct-to-wrong changes and 147 reversals.1

Sources (1)

• [1]Evaluating Exact Output and Checkpoint-State Prediction in Real Programs
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 12:59 PM

  We present a benchmark for predicting final output and checkpoint state from source and input alone.

  It extends CRUXEval-style output prediction with paired shorter- and longer-trace inputs and checkpoints inside and after a loop.

Extractive summary: sentences quoted from the sources.