Code Understanding is a Bottleneck for Coding Agents

Key points

• Repository benchmarks (e.g., SWE-bench) for coding agents often assume that lines of code edited can predict task difficulty, but such datasets' poor control over code and task types makes it hard to know which abilities truly drive agent errors.1
• CABRA builds tasks from scratch as call graph transformations and scales difficulty via a task size parameter on four axes: function traversal, search, runtime resolution, and instruction following.1
• More broadly, we argue for synthetic evaluations like CABRA to unmask LLM weaknesses trivialized by tools (e.g., needle-in-a-haystack) and abilities beyond just editing (e.g., understanding) that coding agents still find difficult, pairing SWE-bench-style tasks with controlled diagnosis.1

Sources (1)

• [1]Code Understanding is a Bottleneck for Coding Agents
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 06:29 AM

  We present CABRA: a Coding Ability Blueprint for Rigorous Agent evaluation.

  Repository benchmarks (e.g., SWE-bench) for coding agents often assume that lines of code edited can predict task difficulty, but such datasets' poor control over code and task types makes it hard to know which abilities truly drive agent errors.

Extractive summary: sentences quoted from the sources.