Input-Blind Controls Produce Substantial Oracle Headroom for Layer Programs in Multiple-Choice Evaluation

Key points

• For layer programs, oracle evaluations use known answers to estimate the potential gain from this flexibility, before a practical selector is available.1
• This study examines this distinction using 32 layer-skipping and repetition programs on two models and 4,413 multiple-choice items.1
• The analysis compares their gains over a fixed action selected without the evaluation prompt with those of input-blind perturbations at the same sites, re-evaluating selections on another prompt.1
• These results show that substantial headroom can persist across prompts with shared option order without establishing a benefit specific to the selected layer computation; neither ordering against these controls identifies that benefit.1

Sources (1)

• [1]Input-Blind Controls Produce Substantial Oracle Headroom for Layer Programs in Multiple-Choice Evaluation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 04:35 PM

  Adaptive computation aims to improve language-model inference by tailoring execution to each input.

  For layer programs, oracle evaluations use known answers to estimate the potential gain from this flexibility, before a practical selector is available.

Extractive summary: sentences quoted from the sources.