SWE-bench Verified
500 real GitHub issues from Python repos, human-validated. The agent edits the repo until the hidden tests pass.
Agent Index weight
8%
in the Agentic coding pillar
Models tested
32
run by Epoch AI on a shared harness
Results
| # | Model | Score | Normalised | Harness | Cost/task | Date |
|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 Supported Anthropic | 83.5% | source | — | Apr 20, 2026 | |
| 2 | GPT-5.5 Supported OpenAI | 80.6% | source | — | Apr 24, 2026 | |
| 3 | Gemini 3.5 Flash Supported Google | 79.3% | source | — | Jun 1, 2026 | |
| 4 | Claude Opus 4.6 Supported Anthropic | 78.7% | source | — | Feb 18, 2026 | |
| 5 | GLM-5.2 Supported Z.ai (Zhipu) | 78.7% | source | — | Jun 25, 2026 | |
| 6 | DeepSeek V4 Pro Estimated DeepSeek | 77.6% | source | — | Jun 18, 2026 | |
| 7 | Qwen3.7 Max Alibaba (Qwen) | 77.3% | source | — | Jun 18, 2026 | |
| 8 | GPT-5.4 Supported OpenAI | 76.9% | source | — | Mar 6, 2026 | |
| 9 | Claude Opus 4.5 (latest) Supported Anthropic | 76.7% | source | — | Feb 5, 2026 | |
| 10 | Kimi K2.6 Estimated Moonshot AI | 76.7% | source | — | May 8, 2026 | |
| 11 | Qwen3.6 Max Preview Estimated Alibaba (Qwen) | 76.7% | source | — | May 28, 2026 | |
| 12 | Gemini 3.1 Pro Preview Supported Google | 75.6% | source | — | Feb 24, 2026 | |
| 13 | Gemini 3 Flash Preview Supported Google | 75.4% | source | — | Feb 18, 2026 | |
| 14 | Claude Sonnet 4.6 Supported Anthropic | 75.2% | source | — | Feb 21, 2026 | |
| 15 | GPT-5.3 Codex Supported OpenAI | 74.8% | source | — | Feb 25, 2026 | |
| 16 | GLM-5.1 Estimated Z.ai (Zhipu) | 74.2% | source | — | May 15, 2026 | |
| 17 | GPT-5.2 Supported OpenAI | 73.8% | source | — | Feb 12, 2026 | |
| 18 | Kimi K2.5 Supported Moonshot AI | 73.8% | source | — | Feb 17, 2026 | |
| 19 | GPT-5 Supported OpenAI | 73.6% | source | — | Feb 6, 2026 | |
| 20 | Claude Opus 4.1 (latest) Supported Anthropic | 73.3% | source | — | Feb 11, 2026 | |
| 21 | Gemini 3 Pro Preview Supported Google | 72.9% | source | — | Feb 13, 2026 | |
| 22 | GLM-5 Supported Z.ai (Zhipu) | 72.1% | source | — | Feb 15, 2026 | |
| 23 | Claude Sonnet 4.5 (latest) Supported Anthropic | 71.3% | source | — | Feb 5, 2026 | |
| 24 | Claude-Opus-4 Supported Anthropic | 70.7% | source | — | Feb 6, 2026 | |
| 25 | GPT-5.1 Supported OpenAI | 68.0% | source | — | Feb 18, 2026 | |
| 26 | GPT-5 Mini Estimated OpenAI | 64.7% | source | — | Feb 1, 2026 | |
| 27 | o3 Supported OpenAI | 62.3% | source | — | Feb 12, 2026 | |
| 28 | Claude 3.7 Sonnet Supported Anthropic | 61.0% | source | — | Feb 4, 2026 | |
| 29 | Qwen3.6 Plus Estimated Alibaba (Qwen) | 57.9% | source | — | May 14, 2026 | |
| 30 | Gemini 2.5 Pro Supported Google | 57.6% | source | — | Feb 13, 2026 | |
| 31 | GPT-4.1 OpenAI | 48.5% | source | — | Feb 8, 2026 | |
| 32 | GPT-4o (2024-08-06) Supported OpenAI | 31.0% | source | — | Feb 11, 2026 |
For models without a SWE-bench Verified result, the index estimates one from Epoch’s Capabilities Index using a logistic fit on 31 models (residual SD 0.22 on the logit scale). Estimates are labelled and widen the model’s range.