Model comparison

Coding starting choices

Compare coding models by score and cost.

Sources & method

DeepSWE v1.1 · PASS@1 · 113 tasks · mini-swe-agent. Captured October 12, 2026; leaderboard updated September 22. Latest model names verified October 12.

Picks use the lowest reported cost within 10 or 3 score points of each cohort’s best. Estimates and reported averages stay separate. Run snapshots, budgets and retries are unspecified.

Costs are reported averages or estimates. Missing measurements stay blank and unplotted. Leaderboard · Methodology · Changelog

All configurations

Configuration

Additional benchmark evidence

Compare results within each benchmark and evaluator. Scores describe agent systems; blank cells mean unavailable.

Full evidence JSON · captured October 13, 2026

Current observations · 73 measured configurations

Terminal-Bench 4.0.0 · Harbor · official board

66 tasks · resolution rate. Harbor and AA use separate evaluations. Version 4.0 is distinct from 2.0/2.1; run dates, builds and budgets/retries are unspecified.

Rounded run totals have unspecified category/attempt coverage. Grok cost covers 324/330 attempts. Muse’s zero sentinel means unavailable.

Terminal-Bench 4.0.0 · Harbor · official board
Model	Agent / harness	Effort	Score	Reported run tokens	Reported run USD	Source
Claude Opus 5.5	Claude Code	max	64.8% ± 3.1 pp	8.0B	$4.7k	Source
Claude Sonnet 5.5	Claude Code	max	61.8% ± 2.9 pp	19.4B	$7.3k	Source
GPT-6 Astra	Codex	max	58.2% ± 2.8 pp	1.5B	$3.3k	Source
GPT-6.1 Sol	Codex	max	58.2% ± 3.1 pp	1.5B	$634.24	Source
Claude Fable 5.1	Claude Code	max	57.9% ± 3.8 pp	2.7B	$6.2k	Source
GPT-6 Astra	Codex	xhigh	57.9% ± 2.7 pp	1.2B	$2.4k	Source
GPT-6 Astra	Codex	high	57.9% ± 3.0 pp	1.2B	$2.3k	Source
Claude Fable 5.1	Claude Code	xhigh	57.9% ± 3.4 pp	2.3B	$4.9k	Source
Claude Fable 5.1	Claude Code	high	54.5% ± 3.4 pp	2.2B	$4.0k	Source
GPT-6 Astra	Codex	medium	54.2% ± 2.7 pp	1.1B	$1.9k	Source
Claude Fable 5.1	Claude Code	medium	53.9% ± 3.4 pp	1.6B	$2.8k	Source
GPT-6 Astra	Codex	low	50.6% ± 2.8 pp	889.8M	$1.6k	Source
Claude Fable 5.1	Claude Code	low	43.3% ± 3.6 pp	1.3B	$2.4k	Source
Grok 4.7	Grok Build	xhigh	37.6% ± 3.5 pp	5.5B	$3.68k (partial: 324/330)	Source
Gemini 3.8 Flash	mini-SWE-agent	high	19.1% ± 3.4 pp	17.2B	$1.8k	Source
GPT-6 Luna	Codex	max	16.4% ± 2.7 pp	3.9B	$72.14	Source
Muse Spark 1.3	Muse Code	xhigh	14.5% ± 2.9 pp	9.0B		Source
Claude Haiku 5.5						Model docs
Gemini 3.5 Flash-Lite						Model docs
Gemini 3.1 Pro preview						Model docs
Muse Wonder						Model docs
Muse Wonder Full						Model docs

Method · captured 2026-10-13; evaluation dates unavailable.

Terminal-Bench 4.0.0 · Artificial Analysis · v1.5

66 tasks · pass@1, averaged over 3 attempts per task. AA and Harbor use separate evaluations. Version 4.0 is distinct from 2.0/2.1; run dates, builds and budgets/retries are unspecified.

Terminal-Bench 4.0.0 · Artificial Analysis · v1.5
Model	Agent / harness	Effort	Score	Source
Claude Sonnet 5.5	Claude Code	max	66.0%	Source
Claude Opus 5.5	Claude Code	max	63.0%	Source
Claude Sonnet 5.5	Claude Code	xhigh	58.0%	Source
Claude Fable 5.1	Claude Code · mixed-model fallback	max	58.0%	Source
Claude Sonnet 5.5	Claude Code	high	42.0%	Source
Claude Sonnet 5.5	Claude Code	medium	27.0%	Source
Claude Sonnet 5.5	Claude Code	low	25.0%	Source
Claude Haiku 5.5	Claude Code	xhigh	24.0%	Source
Claude Haiku 5.5	Claude Code	max	30.0%	Source
Claude Haiku 5.5	Claude Code	high	13.0%	Source
Claude Haiku 5.5	Claude Code	medium	12.0%	Source
Claude Haiku 5.5	Claude Code	low	6.0%	Source
GPT-6.1 Sol	Codex	xhigh	55.0%	Source
GPT-6 Astra	Codex	max	56.0%	Source
GPT-6.1 Sol	Codex	medium	52.0%	Source
GPT-6.1 Sol	Codex	high	50.0%	Source
GPT-6.1 Sol	Codex	max	53.0%	Source
GPT-6.1 Sol	Codex	low	49.0%	Source
GPT-6 Luna	Codex	max	15.0%	Source
Grok 4.7	Grok Build	xhigh	33.0%	Source
Muse Wonder Full	Muse Code	max	59.0%	Source
Muse Wonder	Muse Code	max	47.0%	Source
Muse Spark 1.3	Muse Code	max	32.0%	Source
Muse Spark 1.3	Muse Code	xhigh	17.0%	Source
Gemini 3.5 Flash-Lite				Model docs
Gemini 3.8 Flash				Model docs
Gemini 3.1 Pro preview				Model docs

Method · captured 2026-10-13; evaluation dates unavailable.

SWE-Atlas-QnA · Artificial Analysis · v1.5

124 repository Q&A tasks · pass@1, averaged over 3 attempts per task. AA and Scale use separate evaluations. Pooled efficiency covers 303 tasks; run dates, builds and budgets/retries are unspecified.

SWE-Atlas-QnA · Artificial Analysis · v1.5
Model	Agent / harness	Effort	Score	Source
Claude Sonnet 5.5	Claude Code	max	67.0%	Source
Claude Opus 5.5	Claude Code	max	66.0%	Source
Claude Sonnet 5.5	Claude Code	xhigh	62.0%	Source
Claude Fable 5.1	Claude Code · mixed-model fallback	max	65.0%	Source
Claude Sonnet 5.5	Claude Code	high	56.0%	Source
Claude Sonnet 5.5	Claude Code	medium	45.0%	Source
Claude Sonnet 5.5	Claude Code	low	39.0%	Source
Claude Haiku 5.5	Claude Code	xhigh	52.0%	Source
Claude Haiku 5.5	Claude Code	max	34.0%	Source
Claude Haiku 5.5	Claude Code	high	46.0%	Source
Claude Haiku 5.5	Claude Code	medium	42.0%	Source
Claude Haiku 5.5	Claude Code	low	33.0%	Source
GPT-6.1 Sol	Codex	xhigh	61.0%	Source
GPT-6 Astra	Codex	max	62.0%	Source
GPT-6.1 Sol	Codex	medium	61.0%	Source
GPT-6.1 Sol	Codex	high	60.0%	Source
GPT-6.1 Sol	Codex	max	58.0%	Source
GPT-6.1 Sol	Codex	low	55.0%	Source
GPT-6 Luna	Codex	max	44.0%	Source
Grok 4.7	Grok Build	xhigh	63.0%	Source
Muse Wonder Full	Muse Code	max	68.0%	Source
Muse Wonder	Muse Code	max	67.0%	Source
Muse Spark 1.3	Muse Code	max	59.0%	Source
Muse Spark 1.3	Muse Code	xhigh	55.0%	Source
Gemini 3.5 Flash-Lite				Model docs
Gemini 3.8 Flash				Model docs
Gemini 3.1 Pro preview				Model docs

Method · captured 2026-10-13; evaluation dates unavailable.

SWE-Atlas-QnA · Scale · owner excerpt

124 repository Q&A tasks · Task Resolve Rate requires every rubric criterion to pass. Scale and AA use separate evaluations. Pooled efficiency covers 303 tasks; run dates, builds and budgets/retries are unspecified.

SWE-Atlas-QnA · Scale · owner excerpt
Model	Agent / harness	Effort	Score	Source
Claude Fable 5.1	Claude Code	xhigh	59.95% ± 4.85 pp	Source
GPT-6 Astra	Codex	xhigh	59.14% ± 4.88 pp	Source
Gemini 3.8 Flash	mini-SWE-agent		47.04% ± 5.08 pp	Source
GPT-6 Luna				Model docs
GPT-6.1 Sol				Model docs
Claude Haiku 5.5				Model docs
Claude Sonnet 5.5				Model docs
Claude Opus 5.5				Model docs
Gemini 3.5 Flash-Lite				Model docs
Gemini 3.1 Pro preview				Model docs
Muse Spark 1.3				Model docs
Muse Wonder				Model docs
Muse Wonder Full				Model docs
Grok 4.7				Model docs

Method · captured 2026-10-13; evaluation dates unavailable.

SWE-bench Pro V2 · HARD-51 · Scale · V2 HARD-51

51 HARD tasks · Resolve Rate. The 642-task Full and 731-task v1 evaluations are separate. Row-level builds, budgets, retries and grading protocol are unspecified.

SWE-bench Pro V2 · HARD-51 · Scale · V2 HARD-51
Model	Agent / harness	Effort	Score	Source
Claude Fable 5.1	Claude Code	high	92.2%	Source
GPT-6 Astra	Codex	high	90.2%	Source
Claude Sonnet 5.5	mini-swe-agent	xhigh	70.59%	Source
Gemini 3.8 Flash	mini-swe-agent	high	58.8%	Source
Claude Haiku 5.5	mini-swe-agent	high	52.94% ± 13.01 pp	Source
GPT-6 Luna				Model docs
GPT-6.1 Sol				Model docs
Claude Opus 5.5				Model docs
Gemini 3.5 Flash-Lite				Model docs
Gemini 3.1 Pro preview				Model docs
Muse Spark 1.3				Model docs
Muse Wonder				Model docs
Muse Wonder Full				Model docs
Grok 4.7				Model docs

Method · captured 2026-10-13; evaluation dates unavailable.

Additional benchmark History · 44 observations

Older model versions and settings, with their original measurements.

Additional benchmark History
Original model label	Benchmark	Evaluation source	Agent / harness	Effort	Score	Reported run USD	Source
Opus 5	Terminal-Bench 4.0.0	Harbor · official board	Claude Code	xhigh	53.9% ± 3.2 pp	$6.1k	Source
Opus 5	Terminal-Bench 4.0.0	Harbor · official board	Claude Code	max	51.8% ± 3.4 pp	$6.0k	Source
Opus 5	Terminal-Bench 4.0.0	Harbor · official board	Claude Code	high	50.3% ± 3.7 pp	$4.7k	Source
GPT-6 Sol	Terminal-Bench 4.0.0	Harbor · official board	Codex	max	49.4% ± 3.2 pp	$1.5k	Source
Opus 5	Terminal-Bench 4.0.0	Harbor · official board	Claude Code	medium	44.8% ± 3.8 pp	$3.2k	Source
Fable 5	Terminal-Bench 4.0.0	Harbor · official board	Claude Code	max	44.5% ± 3.8 pp	$7.3k	Source
GLM-5.3	Terminal-Bench 4.0.0	Harbor · official board	Claude Code	max	41.8% ± 3.2 pp	$2.7k	Source
GPT-5.6 Sol	Terminal-Bench 4.0.0	Harbor · official board	Codex	max	37.3% ± 3.8 pp	$2.5k	Source
GLM-5.3-Flash	Terminal-Bench 4.0.0	Harbor · official board	Claude Code	none	35.8% ± 3.5 pp	$2.9k	Source
Opus 5	Terminal-Bench 4.0.0	Harbor · official board	Claude Code	low	34.8% ± 3.9 pp	$2.4k	Source
Qwen3.8-Max-0902	Terminal-Bench 4.0.0	Harbor · official board	Claude Code	max	27.0% ± 3.8 pp	$4.6k	Source
Opus 4.8	Terminal-Bench 4.0.0	Harbor · official board	Claude Code	max	23.6% ± 3.6 pp	$6.5k	Source
GPT-5.6 Terra	Terminal-Bench 4.0.0	Harbor · official board	Codex	max	21.5% ± 3.3 pp	$1.7k	Source
Grok 4.6	Terminal-Bench 4.0.0	Harbor · official board	Grok Build	high	20.3% ± 3.1 pp	$3.6k	Source
GPT-5.6 Luna	Terminal-Bench 4.0.0	Harbor · official board	Codex	max	17.3% ± 2.8 pp	$346.67	Source
Grok 4.5	Terminal-Bench 4.0.0	Harbor · official board	Grok Build	high	12.4% ± 2.6 pp	$2.1k	Source
Sonnet 5	Terminal-Bench 4.0.0	Harbor · official board	Claude Code	max	12.4% ± 3.1 pp	$9.6k	Source
Gemini 3.7 Flash	Terminal-Bench 4.0.0	Harbor · official board	mini-SWE-agent	high	11.2% ± 2.4 pp	$1.3k	Source
Opus 5	Terminal-Bench 4.0.0	Artificial Analysis · v1.5	Claude Code	max	55.0%		Source
Opus 5	SWE-Atlas-QnA	Artificial Analysis · v1.5	Claude Code	max	62.0%		Source
Step 5 Preview	Terminal-Bench 4.0.0	Artificial Analysis · v1.5	Claude Code	unspecified	33.0%		Source
Step 5 Preview	SWE-Atlas-QnA	Artificial Analysis · v1.5	Claude Code	unspecified	58.0%		Source
Qwen3.8 Max	Terminal-Bench 4.0.0	Artificial Analysis · v1.5	Claude Code	unspecified	17.0%		Source
Qwen3.8 Max	SWE-Atlas-QnA	Artificial Analysis · v1.5	Claude Code	unspecified	62.0%		Source
GPT-6 Sol	Terminal-Bench 4.0.0	Artificial Analysis · v1.5	Codex	max	43.0%		Source
GPT-6 Sol	SWE-Atlas-QnA	Artificial Analysis · v1.5	Codex	max	58.0%		Source
GPT-5.6 Sol	Terminal-Bench 4.0.0	Artificial Analysis · v1.5	Codex	max	37.0%		Source
GPT-5.6 Sol	SWE-Atlas-QnA	Artificial Analysis · v1.5	Codex	max	54.0%		Source
GPT-5.6 Luna	Terminal-Bench 4.0.0	Artificial Analysis · v1.5	Codex	max	15.0%		Source
GPT-5.6 Luna	SWE-Atlas-QnA	Artificial Analysis · v1.5	Codex	max	49.0%		Source
DeepSeek V4 Pro 0813	Terminal-Bench 4.0.0	Artificial Analysis · v1.5	Codex	max	10.0%		Source
DeepSeek V4 Pro 0813	SWE-Atlas-QnA	Artificial Analysis · v1.5	Codex	max	62.0%		Source
DeepSeek V4 Flash 0731	Terminal-Bench 4.0.0	Artificial Analysis · v1.5	Codex	max	11.0%		Source
DeepSeek V4 Flash 0731	SWE-Atlas-QnA	Artificial Analysis · v1.5	Codex	max	51.0%		Source
Grok 4.6	Terminal-Bench 4.0.0	Artificial Analysis · v1.5	Grok Build	xhigh	18.0%		Source
Grok 4.6	SWE-Atlas-QnA	Artificial Analysis · v1.5	Grok Build	xhigh	58.0%		Source
Opus 5	SWE-bench Pro V2 · HARD-51	Scale · V2 HARD-51	Claude Code	xhigh	98.0%		Source
Kimi-K3	SWE-bench Pro V2 · HARD-51	Scale · V2 HARD-51	mini-swe-agent	max	88.2%		Source
Sonnet 5	SWE-bench Pro V2 · HARD-51	Scale · V2 HARD-51	Claude Code	xhigh	88.2%		Source
GPT-5.6 Terra	SWE-bench Pro V2 · HARD-51	Scale · V2 HARD-51	Codex	xhigh	86.3%		Source
GLM-5.3	SWE-bench Pro V2 · HARD-51	Scale · V2 HARD-51	mini-swe-agent	max	84.3%		Source
GPT-5.6 Sol	SWE-bench Pro V2 · HARD-51	Scale · V2 HARD-51	Codex	xhigh	82.4%		Source
Inkling	SWE-bench Pro V2 · HARD-51	Scale · V2 HARD-51	mini-swe-agent	xhigh	56.9%		Source
Haiku 4.5	SWE-bench Pro V2 · HARD-51	Scale · V2 HARD-51	Claude Code	xhigh	25.5%		Source

AA suite-wide efficiency · 303 pooled tasks

Averages across 303 tasks from DeepSWE v1.1, Terminal-Bench 4.0 and SWE-Atlas-QnA; not attributable to a single benchmark. Time covers the active agent process. API charges are estimates; token categories, telemetry coverage and pricing dates are unspecified.

AA v1.5 · pooled per-attempt averages across 303 tasks · current and History
Model / original label	Bucket	Agent / harness	Effort	Reported total tokens / attempt	Reported agent time / attempt	Estimated API USD / attempt	Source
Claude Sonnet 5.5	Current	Claude Code	max	27.7M	1.5h	14.19	Source
Claude Opus 5.5	Current	Claude Code	max	15.6M	1.1h	13.04	Source
Claude Sonnet 5.5	Current	Claude Code	xhigh	7.1M	27.0m	3.33	Source
Claude Fable 5.1	Current	Claude Code · mixed-model fallback	max	5.7M	34.8m	12.39	Source
Opus 5	History	Claude Code	max	11.4M	41.9m	10.79	Source
Claude Sonnet 5.5	Current	Claude Code	high	2.7M	12.3m	1.24	Source
Step 5 Preview	History	Claude Code	unspecified	21.7M	56.4m	1.66	Source
Claude Sonnet 5.5	Current	Claude Code	medium	1.3M	8.5m	0.62	Source
Qwen3.8 Max	History	Claude Code	unspecified	8.7M	1.1h	3.48	Source
Claude Sonnet 5.5	Current	Claude Code	low	977.3K	6.3m	0.48	Source
Claude Haiku 5.5	Current	Claude Code	xhigh	7.1M	25.9m	0.61	Source
Claude Haiku 5.5	Current	Claude Code	max	13.8M	1.5h	2.58	Source
Claude Haiku 5.5	Current	Claude Code	high	5.1M	18.4m	0.36	Source
Claude Haiku 5.5	Current	Claude Code	medium	3.5M	14.0m	0.23	Source
Claude Haiku 5.5	Current	Claude Code	low	2.4M	11.6m	0.14	Source
GPT-6.1 Sol	Current	Codex	xhigh	3.2M	15.5m	1.04	Source
GPT-6 Astra	Current	Codex	max	3.3M	29.4m	7.47	Source
GPT-6.1 Sol	Current	Codex	medium	2.3M	10.9m	0.7	Source
GPT-6.1 Sol	Current	Codex	high	2.8M	13.3m	0.89	Source
GPT-6.1 Sol	Current	Codex	max	4M	24.4m	1.55	Source
GPT-6.1 Sol	Current	Codex	low	1.7M	8.6m	0.5	Source
GPT-6 Sol	History	Codex	max	9.8M	22.3m	2.99	Source
GPT-5.6 Sol	History	Codex	max	10.2M	20.6m	6.35	Source
GPT-5.6 Luna	History	Codex	max	14.9M	23.5m	0.44	Source
DeepSeek V4 Pro 0813	History	Codex	max	24.2M	40.3m	0.24	Source
GPT-6 Luna	Current	Codex	max	10.2M	21.4m	0.18	Source
DeepSeek V4 Flash 0731	History	Codex	max	15.4M	18.3m	0.09	Source
Grok 4.7	Current	Grok Build	xhigh	14.3M	39.2m	8.82	Source
Grok 4.6	History	Grok Build	xhigh	5.5M	19.5m	3.57	Source
Muse Wonder Full	Current	Muse Code	max	15M	28.8m		Source
Muse Wonder	Current	Muse Code	max	15M	22.4m	8.05	Source
Muse Spark 1.3	Current	Muse Code	max	16.7M	18.4m	3.98	Source
Muse Spark 1.3	Current	Muse Code	xhigh	16.2M	18.3m	3.47	Source

DeepSWE evidence tables

How to read the tables

Blank cells mean unavailable. Versions retain their own measurements. Serving snapshots, run dates, harness builds, budgets, retries and speed tiers are unspecified; effort labels are not a common compute scale.

Latest model names, IDs and availability

Latest family versions · verified October 12, 2026
Official name	API ID	Release date	Availability	Source
GPT-6 Luna	gpt-6-luna		Documented API model; Work/Codex access depends on rollout/client/workspace.	Official
GPT-6.1 Sol	gpt-6.1-sol		Documented API model; Work/Codex access depends on rollout/client/workspace.	Official
GPT-6 Astra	gpt-6-astra		Documented API model.	Official
Claude Haiku 5.5	claude-haiku-5-5	2026-10-07	Released on Claude Platform and partners.	Official
Claude Sonnet 5.5	claude-sonnet-5-5	2026-09-28	Released on Claude Platform and partners.	Official
Claude Opus 5.5	claude-opus-5-5	2026-09-22	Released on Claude Platform and partners.	Official
Claude Fable 5.1	claude-fable-5-1		Current documented model for demanding reasoning.	Official
Gemini 3.5 Flash-Lite	gemini-3.5-flash-lite		Stable text-output API model; excludes TTS/image variants.	Official
Gemini 3.8 Flash	gemini-3.8-flash		Stable text-output API model; excludes TTS/image/live variants.	Official
Gemini 3.1 Pro preview	gemini-3.1-pro-preview		Preview. Board label gemini-3.1-pro does not establish this exact endpoint/snapshot.	Official
Muse Spark 1.3		2026-09-02	Meta Model API and Muse Code; exact API ID unverified.	Official
Muse Wonder		2026-10-12	Meta Model API and Muse Code; exact API ID unverified.	Official
Muse Wonder Full		2026-10-12	Muse Code subscribers; public API access/ID unverified.	Official
Grok 4.7	grok-4.7	2026-09-21	Documented API flagship; preserve xAI/Grok user label.	Official

Meta API IDs are unverified. The board’s Gemini 3.1 Pro label is not matched to the exact preview endpoint. Restricted Gemini 4 Argon is outside the current graph.

Benchmark score

DeepSWE v1.1 · PASS@1 · 113 tasks
Latest model	Effort	Score (%)	Interval (± pp)	Source
GPT-6 Luna				DeepSWE
GPT-6.1 Sol				DeepSWE
GPT-6 Astra	xhigh	74	3	DeepSWE
GPT-6 Astra	high	73	3	DeepSWE
GPT-6 Astra	max	73	1	DeepSWE
GPT-6 Astra	medium	73	3	DeepSWE
GPT-6 Astra	low	67	1	DeepSWE
Claude Haiku 5.5				DeepSWE
Claude Sonnet 5.5				DeepSWE
Claude Opus 5.5				DeepSWE
Claude Fable 5.1				DeepSWE
Gemini 3.5 Flash-Lite				DeepSWE
Gemini 3.8 Flash	high	74	1	DeepSWE
Gemini 3.8 Flash	medium	71	2	DeepSWE
Gemini 3.1 Pro preview				DeepSWE
Muse Spark 1.3				DeepSWE
Muse Wonder				DeepSWE
Muse Wonder Full				DeepSWE
Grok 4.7				DeepSWE

Tokens

Rounded average output tokens from the leaderboard. Input, cache and reasoning counters are unavailable.

Leaderboard token evidence · counts; rounded average output shown separately
Latest model	Effort	Avg output (rounded)	Source
GPT-6 Luna			DeepSWE
GPT-6.1 Sol			DeepSWE
GPT-6 Astra	xhigh	30k	DeepSWE
GPT-6 Astra	high	27k	DeepSWE
GPT-6 Astra	max	61k	DeepSWE
GPT-6 Astra	medium	20k	DeepSWE
GPT-6 Astra	low	11k	DeepSWE
Claude Haiku 5.5			DeepSWE
Claude Sonnet 5.5			DeepSWE
Claude Opus 5.5			DeepSWE
Claude Fable 5.1			DeepSWE
Gemini 3.5 Flash-Lite			DeepSWE
Gemini 3.8 Flash	high	143k	DeepSWE
Gemini 3.8 Flash	medium	125k	DeepSWE
Gemini 3.1 Pro preview			DeepSWE
Muse Spark 1.3			DeepSWE
Muse Wonder			DeepSWE
Muse Wonder Full			DeepSWE
Grok 4.7			DeepSWE

Time

Leaderboard duration · no aggregate duration supplied
Latest model	Effort	Source
GPT-6 Luna		DeepSWE
GPT-6.1 Sol		DeepSWE
GPT-6 Astra	xhigh	DeepSWE
GPT-6 Astra	high	DeepSWE
GPT-6 Astra	max	DeepSWE
GPT-6 Astra	medium	DeepSWE
GPT-6 Astra	low	DeepSWE
Claude Haiku 5.5		DeepSWE
Claude Sonnet 5.5		DeepSWE
Claude Opus 5.5		DeepSWE
Claude Fable 5.1		DeepSWE
Gemini 3.5 Flash-Lite		DeepSWE
Gemini 3.8 Flash	high	DeepSWE
Gemini 3.8 Flash	medium	DeepSWE
Gemini 3.1 Pro preview		DeepSWE
Muse Spark 1.3		DeepSWE
Muse Wonder		DeepSWE
Muse Wonder Full		DeepSWE
Grok 4.7		DeepSWE

Steps

Leaderboard agent steps · rounded average per task/trial
Latest model	Effort	Average steps	Source
GPT-6 Luna			DeepSWE
GPT-6.1 Sol			DeepSWE
GPT-6 Astra	xhigh	29	DeepSWE
GPT-6 Astra	high	27	DeepSWE
GPT-6 Astra	max	28	DeepSWE
GPT-6 Astra	medium	26	DeepSWE
GPT-6 Astra	low	20	DeepSWE
Claude Haiku 5.5			DeepSWE
Claude Sonnet 5.5			DeepSWE
Claude Opus 5.5			DeepSWE
Claude Fable 5.1			DeepSWE
Gemini 3.5 Flash-Lite			DeepSWE
Gemini 3.8 Flash	high	166	DeepSWE
Gemini 3.8 Flash	medium	147	DeepSWE
Gemini 3.1 Pro preview			DeepSWE
Muse Spark 1.3			DeepSWE
Muse Wonder			DeepSWE
Muse Wonder Full			DeepSWE
Grok 4.7			DeepSWE

Cost

Astra uses expected launch-rate estimates dated September 3. Other reported cost calculations are not fully verified. Billed totals, complete benchmark cost and retry-inclusive cost per successful completion remain blank. Pricing and reporting changes

Leaderboard cost · USD · estimates and reported averages
Latest model	Effort	Reported avg / attempt	Basis / effective date	Source
GPT-6 Luna				DeepSWE
GPT-6.1 Sol				DeepSWE
GPT-6 Astra	xhigh	$4.43	Expected launch-rate estimate · 2026-09-03	DeepSWE
GPT-6 Astra	high	$3.92	Expected launch-rate estimate · 2026-09-03	DeepSWE
GPT-6 Astra	max	$7.50	Expected launch-rate estimate · 2026-09-03	DeepSWE
GPT-6 Astra	medium	$3.08	Expected launch-rate estimate · 2026-09-03	DeepSWE
GPT-6 Astra	low	$1.60	Expected launch-rate estimate · 2026-09-03	DeepSWE
Claude Haiku 5.5				DeepSWE
Claude Sonnet 5.5				DeepSWE
Claude Opus 5.5				DeepSWE
Claude Fable 5.1				DeepSWE
Gemini 3.5 Flash-Lite				DeepSWE
Gemini 3.8 Flash	high	$2.36	Source-reported; pricing basis unverified	DeepSWE
Gemini 3.8 Flash	medium	$1.97	Source-reported; pricing basis unverified	DeepSWE
Gemini 3.1 Pro preview				DeepSWE
Muse Spark 1.3				DeepSWE
Muse Wonder				DeepSWE
Muse Wonder Full				DeepSWE
Grok 4.7				DeepSWE

One task, two trials

happy-dom-abort-pending-body-reads · DeepSWE v1.1. These are individual inspected trials, separate from leaderboard averages. Both report a pass; equal “high” settings do not establish equivalent budgets.

Single task · source-reported trial outcome
Model	Effort	Outcome	Fail → pass tests	Pass → pass tests	Run started (UTC)	Source
GPT-6 Astra	high	pass			2026-08-31T23:15:39Z	Trial
Gemini 3.8 Flash	high	pass	14/14	165/165	2026-08-29T08:25:20.393198Z	Trial

Single-trial token counters · source-reported telemetry
Model	Effort	Input incl. cache	Uncached input	Cache read	Cache write	Output	Reasoning	Source
GPT-6 Astra	high	2,645,526	987	2,528,884	115,655	28,879	11,206	Trial
Gemini 3.8 Flash	high	47,636,992		47,374,336		221,856		Trial

Input includes cached tokens. Output and reasoning may overlap, so no combined total is shown. Astra’s trajectory, verifier and patch tabs are disabled; Gemini’s raw trajectory is unreviewed.

Single-trial time and steps · minutes / count
Model	Effort	Trial (min)	Agent (min)	Agent steps	Source
GPT-6 Astra	high	18.3	18.3	37	Trial
Gemini 3.8 Flash	high	19.4	18.5	247	Trial

Single-trial cost · USD · no invoice or complete attributable total
Model	Effort	Reported cost	Billed cost	Completion total	Basis / effective date	Source
GPT-6 Astra	high	$5.43			Expected launch-rate estimate · 2026-09-03	Trial
Gemini 3.8 Flash	high	$4.58			Source-reported; pricing basis unverified	Trial

Task and evaluation conditions

Base commit: 82a0888cb2c87a6123e05424b528f8e8c9b3e426. Harness: mini-swe-agent / Pier / Modal. Declared agent timeout: 5,400 seconds; verifier timeout: 1,800 seconds. Enforced token budget, retry policy, exact harness build and speed tier are unknown.

Trial IDs: happy-dom-abort-pending-body-rea__5merruF and happy-dom-abort-pending-body-rea__8DdxqX3. Astra metrics come from the verified OpenAI handoff; Gemini metrics come from the Vertex AI trial summary. Reported costs are not actual invoices or complete completion costs.

History

Older versions and unmatched records. Datacurve: updated September 22, captured October 12.

Older versions · 43 configurations

Datacurve leaderboard archive · DeepSWE v1.1 · same captured source
Exact source label	Effort	Score (%)	Interval (± pp)	Avg output (rounded)	Avg steps	Avg USD / attempt	Basis / effective date	Source
claude-opus-5	max	74	4	118k	99	$11.84	Source-reported; pricing basis unverified	DeepSWE
claude-opus-5	xhigh	73	3	92k	89	$9.07	Source-reported; pricing basis unverified	DeepSWE
claude-opus-5	high	73	2	64k	73	$6.08	Source-reported; pricing basis unverified	DeepSWE
gpt-5.6-sol	max	73	3	60k	61	$6.46	Source-reported; pricing basis unverified	DeepSWE
gpt-5.6-sol	xhigh	71	1	41k	44	$3.60	Source-reported; pricing basis unverified	DeepSWE
claude-fable-5	xhigh	70	3	80k	68	$13.41	Source-reported; pricing basis unverified	DeepSWE
claude-fable-5	max	70	4	119k	88	$21.63	Source-reported; pricing basis unverified	DeepSWE
gpt-5.6-sol	high	69	1	28k	37	$2.66	Source-reported; pricing basis unverified	DeepSWE
claude-opus-5	medium	69	1	37k	52	$3.29	Source-reported; pricing basis unverified	DeepSWE
claude-fable-5	high	69	1	57k	59	$9.18	Source-reported; pricing basis unverified	DeepSWE
grok-4.6	medium	67	2	50k	70	$3.45	Source-reported; pricing basis unverified	DeepSWE
gpt-5.6-luna	max	67	4	73k	102	$0.61	Source-reported; pricing basis unverified	DeepSWE
grok-4.6	xhigh	67	2	71k	87	$5.50	Source-reported; pricing basis unverified	DeepSWE
gemini-3.7-flash	medium	65	3	94k	117	$2.03	Source-reported; pricing basis unverified	DeepSWE
claude-fable-5	medium	65	4	40k	48	$6.09	Source-reported; pricing basis unverified	DeepSWE
gemini-3.7-flash	high	65	2	107k	125	$2.18	Source-reported; pricing basis unverified	DeepSWE
grok-4.6	high	65	2	61k	79	$4.38	Source-reported; pricing basis unverified	DeepSWE
gpt-5.6-sol	medium	61	2	18k	31	$1.42	Source-reported; pricing basis unverified	DeepSWE
claude-fable-5	low	60	3	25k	38	$3.76	Source-reported; pricing basis unverified	DeepSWE
claude-opus-4.8	max	59	2	135k	120	$13.22	Source-reported; pricing basis unverified	DeepSWE
claude-opus-5	low	58	2	20k	36	$1.66	Source-reported; pricing basis unverified	DeepSWE
gpt-5.6-luna	xhigh	57	2	45k	71	$0.31	Source-reported; pricing basis unverified	DeepSWE
muse-spark-1.2	xhigh	55	2	99k	101	$3.70	Source-reported; pricing basis unverified	DeepSWE
claude-opus-4.8	xhigh	54	4	86k	95	$8.01	Source-reported; pricing basis unverified	DeepSWE
claude-sonnet-5	max	54	4	214k	268	$26.40	Source-reported; pricing basis unverified	DeepSWE
gemini-3.7-flash	low	54	3	73k	130	$1.83	Source-reported; pricing basis unverified	DeepSWE
grok-4.5	high	54	2	36k	61	$2.42	Source-reported; pricing basis unverified	DeepSWE
muse-spark-1.1	xhigh	53	3	74k	96	$2.36	Source-reported; pricing basis unverified	DeepSWE
claude-opus-4.8	high	52	5	50k	73	$4.28	Source-reported; pricing basis unverified	DeepSWE
claude-sonnet-5	xhigh	50	3	121k	186	$11.89	Source-reported; pricing basis unverified	DeepSWE
claude-opus-4.8	medium	49	2	41k	66	$3.44	Source-reported; pricing basis unverified	DeepSWE
claude-sonnet-5	high	48	5	87k	147	$7.43	Source-reported; pricing basis unverified	DeepSWE
gemini-3.6-flash	high	47	4	96k	117	$2.21	Source-reported; pricing basis unverified	DeepSWE
gpt-5.6-sol	low	45	2	11k	23	$0.82	Source-reported; pricing basis unverified	DeepSWE
gpt-5.6-luna	high	44	3	26k	49	$0.16	Source-reported; pricing basis unverified	DeepSWE
grok-4.6	low	42	2	16k	44	$1.04	Source-reported; pricing basis unverified	DeepSWE
claude-opus-4.8	low	41	1	29k	54	$2.29	Source-reported; pricing basis unverified	DeepSWE
claude-sonnet-5	medium	40	3	57k	108	$4.08	Source-reported; pricing basis unverified	DeepSWE
gemini-3.5-flash	high	36	4	76k	105	$3.45	Source-reported; pricing basis unverified	DeepSWE
claude-sonnet-5	low	31	1	36k	77	$2.19	Source-reported; pricing basis unverified	DeepSWE
claude-sonnet-4.6	high	30	4	76k	134	$5.52	Source-reported; pricing basis unverified	DeepSWE
gpt-5.6-luna	medium	11	1	8.2k	24	$0.04	Source-reported; pricing basis unverified	DeepSWE
gpt-5.6-luna	low	2	1	3.1k	12	$0.01	Source-reported; pricing basis unverified	DeepSWE

Unmatched source records · 20 configurations

Latest-version status or endpoint identity is unspecified for these records.

Datacurve leaderboard archive · DeepSWE v1.1 · same captured source
Exact source label	Effort	Score (%)	Interval (± pp)	Avg output (rounded)	Avg steps	Avg USD / attempt	Basis / effective date	Source
gpt-5.6-terra	max	70	3	72k	76	$3.96	Source-reported; pricing basis unverified	DeepSWE
glm-5.3	max	69	3	80k	124	$3.99	Source-reported; pricing basis unverified	DeepSWE
kimi-k3	max	69	5	81k	98	$4.65	Source-reported; pricing basis unverified	DeepSWE
gpt-5.5	xhigh	67	6	46k	82	$7.23	Source-reported; pricing basis unverified	DeepSWE
gpt-5.5	high	64	3	31k	62	$5.10	Source-reported; pricing basis unverified	DeepSWE
glm-5.3-flash	max	63	4	73k	123	$0.24	Source-reported; pricing basis unverified	DeepSWE
deepseek-v4-pro	max	63	6	106k	155	$1.67	Source-reported; pricing basis unverified	DeepSWE
gpt-5.6-terra	xhigh	60	2	40k	43	$1.70	Source-reported; pricing basis unverified	DeepSWE
qwen3.8-max	xhigh	57	3	95k	111	$3.73	Source-reported; pricing basis unverified	DeepSWE
gpt-5.5	medium	54	3	20k	46	$2.75	Source-reported; pricing basis unverified	DeepSWE
gpt-5.6-terra	high	54	4	22k	34	$0.91	Source-reported; pricing basis unverified	DeepSWE
deepseek-v4-flash	max	53	4	108k	153	$0.46	Source-reported; pricing basis unverified	DeepSWE
gpt-5.4	xhigh	52	2	71k	70	$5.65	Source-reported; pricing basis unverified	DeepSWE
glm-5.2	max	44	2	78k	129	$3.92	Source-reported; pricing basis unverified	DeepSWE
glm-5.2	high	36	5	54k	122	$2.84	Source-reported; pricing basis unverified	DeepSWE
gpt-5.6-terra	medium	35	3	12k	25	$0.47	Source-reported; pricing basis unverified	DeepSWE
kimi-k2.7-code		31	1	59k	149	$2.82	Source-reported; pricing basis unverified	DeepSWE
gpt-5.5	low	27	2	9.4k	28	$1.20	Source-reported; pricing basis unverified	DeepSWE
gpt-5.6-terra	low	24	1	8.6k	21	$0.34	Source-reported; pricing basis unverified	DeepSWE
gemini-3.1-pro	high	12	1	28k	76	$2.14	Source-reported; pricing basis unverified	DeepSWE

Original OpenAI launch data · 15 configurations

September 22 / 29 launch charts, checked October 1 and rechecked October 13. Shown in the launch view, separately from Datacurve measurements.

OpenAI launch snapshot · September 22 / 29 · checked October 1, 2026
Model	Effort	Score (%)	USD / attempted task	Source
GPT-6.1 Sol	low	64.38	$0.1714	OpenAI launch
GPT-6.1 Sol	medium	73.01	$0.4196	OpenAI launch
GPT-6.1 Sol	high	75.22	$0.6461	OpenAI launch
GPT-6.1 Sol	xhigh	71.90	$0.7886	OpenAI launch
GPT-6.1 Sol	max	71.90	$1.5711	OpenAI launch
GPT-6 Luna	low	2.43	$0.0057	OpenAI launch
GPT-6 Luna	medium	44.47	$0.0518	OpenAI launch
GPT-6 Luna	high	59.29	$0.0838	OpenAI launch
GPT-6 Luna	xhigh	61.28	$0.1096	OpenAI launch
GPT-6 Luna	max	66.59	$0.2169	OpenAI launch
GPT-6 Astra	low	67.04	$1.5952	OpenAI launch
GPT-6 Astra	medium	72.79	$3.0755	OpenAI launch
GPT-6 Astra	high	73.23	$3.9237	OpenAI launch
GPT-6 Astra	xhigh	74.12	$4.4291	OpenAI launch
GPT-6 Astra	max	73.23	$7.4978	OpenAI launch

Full original dataset and dated history preserved: October 1 snapshot · Earlier snapshot

Coding starting choices

Sources & method

DeepSWE v1.1 · PASS@1 · 113 tasks · mini-swe-agent. Captured October 12, 2026; leaderboard updated September 22. Latest model names verified October 12.

Picks use the lowest reported cost within 10 or 3 score points of each cohort’s best. Estimates and reported averages stay separate. Run snapshots, budgets and retries are unspecified.

Costs are reported averages or estimates. Missing measurements stay blank and unplotted. Leaderboard · Methodology · Changelog

All configurations

Configuration

Matched latest-model measurements and complete attributable usage/cost evidence