How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?
In this paper we find that, under an equal time budget and the same frontier LLM backbone, open-source state-of-the-art harnesses provide no advantages over a single session of a minimal-harness coding agent baseline, pointing to the backbone as the primary driver for performance.
Key points
- Recent autonomous machine learning engineering (MLE) agents have made significant progress on public leaderboards.
- Often motivated by progress stagnation over long-horizon cycles and limited Large Language Model (LLM) primitives, modern MLE agents are deployed on top of increasingly elaborate machinery: multi-agent orchestrators, dedicated retrieval subagents, and more.
- Via a series of large-scale systematic ablation studies, we argue that the machinery layers become redundant in the coding agent setting.
- SCLATE: A Substrate for Continual-Learning Agent Training and Evaluation
Sources (1)
- [1]How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?Apple Machine Learning Research · Oct 1, 12:00 AM
In this paper we find that, under an equal time budget and the same frontier LLM backbone, open-source state-of-the-art harnesses provide no advantages over a single session of a minimal-harness coding agent baseline, pointing to the backbone as the primary driver for performance.
Recent autonomous machine learning engineering (MLE) agents have made significant progress on public leaderboards.
Extractive summary: sentences quoted from the sources.
Before this
- Sep 25, 2026ollama/ollama v0.40.0
- Sep 5, 2026ollama/ollama v0.34.0
- Sep 2, 2026pytorch/pytorch v2.14.0: PyTorch 2.14.0 Release
- Aug 14, 2026ollama/ollama v0.32.12
- Jul 15, 2026huggingface/transformers v5.14.0: Release v5.14.0
- Jul 8, 2026pytorch/pytorch v2.13.0: PyTorch 2.13.0 Release