On the estimation and validity of AI time horizons---a statistical look at the METR plot

Key points

• METR's 50% time horizon measures the human completion time of software tasks that an AI solves with 50% probability, allowing AI capabilities to be expressed in interpretable units.1
• Our fitted spline can be interpreted as a function that converts human time to AI difficulty; it is nearly flat in a region from 2--30 min but close to linear elsewhere.1
• Hence, a time-horizon jump from 3 min to 30 min is much easier than one from 30 min to 5 hours despite the same multiplier of $10 \times$.1
• We suggest that time horizons be interpreted together with the diagnostic plots, especially as new time-horizon-based benchmarks are proposed or existing ones grow to include longer tasks.1

Sources (1)

• [1]On the estimation and validity of AI time horizons---a statistical look at the METR plot
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 05:59 PM

  On 228 tasks and 26 AIs, we recompute the time horizons using splines and item-response theory to relax the assumption that the AI difficulty of a task depends linearly on the log of human time.

  METR's 50% time horizon measures the human completion time of software tasks that an AI solves with 50% probability, allowing AI capabilities to be expressed in interpretable units.

Extractive summary: sentences quoted from the sources.