SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Source

• Person key: ysymyth
• Source kind: paper
• Canonical URL: https://arxiv.org/abs/2310.06770
• License: NOASSERTION
• Public handling: public-metadata-summary-hash-link-only
• Semantic hash: ef84db6f6505acff6015433055cd24bfc9b1d64b3200e7f81a859194b7ff3b99
• First seen: 2026-05-16
• Last changed: 2026-05-16
• Identity guard: Do not confuse with yao-shunyu-alfred, the physics-to-AI researcher at alfredyao.github.io.

Classification

• Category: Software engineering agents and benchmarks
• Topic hub: shunyu-yao-public-corpora
• Project taxonomy: shunyu-yao-project-taxonomy
• Paper map: shunyu-yao-paper-map

Summary

Language models have outpaced our ability to evaluate them effectively, but for their future development it is essential to study the frontier of their capabilities. We find real-world software engineering to be a rich, sustainable, and challenging testbed for evaluating the next generation of language models. To this end, we introduce SWE-bench, an evaluation framework consisting of $2,294$ software engineering problems drawn from real GitHub issues and corresponding pull requests across $12$ popular Python reposi…

What This Teaches

How coding agents are benchmarked, scaffolded, and constrained against real repositories and issues.

Related

• shunyu-yao-public-corpora
• 2026-05-16-yao-shunyu-public-corpora

Public Handling Notes

• EXTRACTED: Metadata and links are from public sources.
• INFERRED: Unclear-license full text, PDFs, and source code should not be mirrored into public wiki pages.
• AMBIGUOUS: Items discovered by keyword search are still separated by source identity and category heuristics.