Benchmarks › DeepSWE

DeepSWE

By Datacurve ↗Since 2026
Complete long software engineering tasks in open-source repositories
74.1top score · GPT-6 Astra
44models ranked
16open-weights models
Pass@1: scored rollout attempt pass ratehigher is better
Oct 23, 2026last updated

How the test works

1

Task

113 tasks

  • What the model doesComplete long software engineering tasks in open-source repositories
  • Kinds of tasksLong-horizon software engineering
  • DomainsTypeScript, Go, Python, JavaScript, Rust
2

Run

4 runs per task

  • Agent / harnessmini-swe-agent
  • How runs combinePass@1 (solved on the first try) is attempt pass rate over scored rollout attempts
3

Score

Pass@1 (solved on the first try): scored rollout attempt pass rate

  • Graded byBehavioral program verifiers
  • Better isHigher
  • Test setAgent cannot access held-out tests or reference solution
Sources: github.com, deepswe.datacurve.ai

DeepSWE 1.1 leaderboardtop 10 of 44

Full leaderboard with filters →
ModelDeepSWE 1.1Per taskPriceSpeedSupportsContext
GPT-6 AstraOpenAI
74.1xhigh
$2.31$10 / $50611.05M
Gemini 3.8 FlashGoogle
73.8high
$1.24$0.75 / $3.751491M
Claude Opus 5Anthropic
73.6max
$5.86$5 / $25551M
GPT-5.6 SolOpenAI
72.7max
$1.99$4 / $201141.05M
Claude Opus 5.5Anthropic
72.3max
—$4 / $20—1M
GPT-6.1 SolOpenAI
72.3max
—$2 / $10—1.05M
Claude Sonnet 5.5Anthropic
70.5max
—$2 / $10—1M
GPT-6 SolOpenAI
70.5max
—$2 / $10—1.05M
Claude Fable 5Anthropic
69.9xhigh
—$10 / $50—1M
GPT-5.6 TerraOpenAI
69.6max
$1.40$2 / $121421.05M

Versions and boards on Model Menu

BoardModels
DeepSWE 1.1Shown above
44
DeepSWE 1
20

DeepSWE questions

What is DeepSWE?›

Complete long software engineering tasks in open-source repositories. It has 113 tasks. Created by Datacurve.

Which model is best on DeepSWE?›

GPT-6 Astra leads DeepSWE 1.1 with 74.1, ahead of Gemini 3.8 Flash (73.8) and Claude Opus 5 (73.6).

What is the best open-weights model on DeepSWE?›

GLM 5.3 with 69, then Kimi K3 (68.5).

How is DeepSWE scored?›

Pass@1 (solved on the first try): scored rollout attempt pass rate; behavioral program verifiers; 4 runs per task per task.

Which versions of DeepSWE are there?›

DeepSWE 1.1 (44 models), DeepSWE 1 (20 models).