AgentSWE: Can Coding Agents Build the Agent You Actually Want?
We introduce AgentSWE, a benchmark for agent software engineering spanning three stages of the agent development lifecycle: Creation of a complete agent from a natural-language requirement, Editing of production agent codebases, and Optimization toward a measurable target.
ProofPaper ↗
Key points
- As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their users' own workflows.
- This delegation carries a quiet assumption, that the delivered agent will faithfully serve the requirement it was built for, rather than merely appear to.
- Evaluating a range of coding agents across models and harnesses, we find that none comes close to delivering what a requirement asks for: the gap between what a builder demonstrates and what it delivers is the norm, not the corner case.
- An accompanying empirical analysis of the failure trajectories identifies recurring failure modes, among them overfitting to the visible cases, hollow runtimes behind code that looks complete, and unsupported claims of completion, each a signal by which coding agents can learn to build the agents they are asked for.
Sources (1)
- [1]AgentSWE: Can Coding Agents Build the Agent You Actually Want?arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 10:35 AM
We introduce AgentSWE, a benchmark for agent software engineering spanning three stages of the agent development lifecycle: Creation of a complete agent from a natural-language requirement, Editing of production agent codebases, and Optimization toward a measurable target.
As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their users' own workflows.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 9, 2026Surfing the Activation Wave: Analyzing the Information and Geometry of LLM Hidden States for Speculative Decoding
- Oct 9, 2026Collaborative Theory of Mind: Learning to Self-Revise by Anticipating User Responses
- Oct 9, 2026Learning Representations to Guide Pixel Diffusion Transformers
- Oct 9, 2026Rethinking Relative Policy Differences Transfer of On-Policy Distillation
- Oct 8, 2026MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis
- Oct 8, 2026Contextual Correction of Parallel Draftsfor Speculative Decoding
