Not All Correct Fixes Run Alike: An Empirical Study of Patch Performance in Repository-Level Issue Resolution

Key points

• Evaluating patch performance is essential but challenging since designing performance workloads for each benchmark instance is costly.1
• Overall, we choose four frontier models, Claude Opus 5, GPT 5.6, DeepSeek V4 Pro, and Qwen 3.8 Max, paired with SWE-agent to resolve the pre-filtered measurable corpus from verified splits of SWE-bench and SWE-bench-Live, yielding 262 cases with at least two different correct solutions, including the developer ground-truth patch.1
• We find that this methodology exhibits notable discrimination capability in runtime performance and identifies performance differences among patches in almost half of the instances.1
• Our empirical study shows that the runtime performance of generated patches is independent of the issue resolution rate of LLMs, with nearly reversed model rankings.1

Sources (1)

• [1]Not All Correct Fixes Run Alike: An Empirical Study of Patch Performance in Repository-Level Issue Resolution
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 01:17 PM

  Repository-level issue-resolution benchmarks such as SWE-bench evaluate submitted patches primarily based on whether they pass the correctness tests, leaving the runtime performance of different test-passing patches largely unexplored.

  Evaluating patch performance is essential but challenging since designing performance workloads for each benchmark instance is costly.

Extractive summary: sentences quoted from the sources.