Aviral GargModel comparison
Latest measured models
Coding benchmark scores and source-specific usage/cost evidence.
Sources & method
DeepSWE v1.1 · PASS@1 · 113 tasks · mini-swe-agent. Captured October 12, 2026; leaderboard updated September 22. Latest model names verified October 12.
Starting picks are the cheapest reported configuration within 10 or 3 score points of that cohort’s best. Expected-rate estimates and other reported averages are separate cohorts. Exact run snapshots, budgets and retries are unresolved; these are source-specific benchmark heuristics.
Costs are reported averages or estimates, not invoices. Missing latest measurements are blank in the tables and have no plotted proxy. Leaderboard · Methodology · Changelog
All configurations
Additional benchmark evidence
Each view keeps its benchmark and evaluator separate. Scores describe agent systems; a controlled five-provider model-only cohort is not established. Blank metrics are unavailable.
Full evidence JSON · captured October 13, 2026
Current observations · 73 measured configurations
Terminal-Bench 4.0.0 · Harbor · official board
66 tasks · resolution rate. Exact run dates, builds and row-specific budgets/retries are unresolved. Different agents are system comparisons. Do not merge official Harbor scores with AA scores; 4.0 differs from2.0/2.1.
Tokens and costs below are rounded run displays, with unresolved category/attempt coverage. Grok cost is partial (324/330 attempts). Muse’s zero sentinel is unavailable, not free. These are not completion invoices.
| Model | Agent / harness | Effort | Score | Reported run tokens | Reported run USD | Input tokens | Output tokens | Time | Attributable completion USD | Actual billed USD | Source |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 | Claude Code | max | 64.8% ± 3.1 pp | 8.0B | $4.7k | Source | |||||
| Claude Sonnet 5.5 | Claude Code | max | 61.8% ± 2.9 pp | 19.4B | $7.3k | Source | |||||
| GPT-6 Astra | Codex | max | 58.2% ± 2.8 pp | 1.5B | $3.3k | Source | |||||
| GPT-6.1 Sol | Codex | max | 58.2% ± 3.1 pp | 1.5B | $634.24 | Source | |||||
| Claude Fable 5.1 | Claude Code | max | 57.9% ± 3.8 pp | 2.7B | $6.2k | Source | |||||
| GPT-6 Astra | Codex | xhigh | 57.9% ± 2.7 pp | 1.2B | $2.4k | Source | |||||
| GPT-6 Astra | Codex | high | 57.9% ± 3.0 pp | 1.2B | $2.3k | Source | |||||
| Claude Fable 5.1 | Claude Code | xhigh | 57.9% ± 3.4 pp | 2.3B | $4.9k | Source | |||||
| Claude Fable 5.1 | Claude Code | high | 54.5% ± 3.4 pp | 2.2B | $4.0k | Source | |||||
| GPT-6 Astra | Codex | medium | 54.2% ± 2.7 pp | 1.1B | $1.9k | Source | |||||
| Claude Fable 5.1 | Claude Code | medium | 53.9% ± 3.4 pp | 1.6B | $2.8k | Source | |||||
| GPT-6 Astra | Codex | low | 50.6% ± 2.8 pp | 889.8M | $1.6k | Source | |||||
| Claude Fable 5.1 | Claude Code | low | 43.3% ± 3.6 pp | 1.3B | $2.4k | Source | |||||
| Grok 4.7 | Grok Build | xhigh | 37.6% ± 3.5 pp | 5.5B | $3.68k (partial: 324/330) | Source | |||||
| Gemini 3.8 Flash | mini-SWE-agent | high | 19.1% ± 3.4 pp | 17.2B | $1.8k | Source | |||||
| GPT-6 Luna | Codex | max | 16.4% ± 2.7 pp | 3.9B | $72.14 | Source | |||||
| Muse Spark 1.3 | Muse Code | xhigh | 14.5% ± 2.9 pp | 9.0B | Source | ||||||
| Claude Haiku 5.5 | Model docs | ||||||||||
| Gemini 3.5 Flash-Lite | Model docs | ||||||||||
| Gemini 3.1 Pro preview | Model docs | ||||||||||
| Muse Wonder | Model docs | ||||||||||
| Muse Wonder Full | Model docs |
Method · captured 2026-10-13; evaluation dates unavailable.
Terminal-Bench 4.0.0 · Artificial Analysis · v1.5
66 tasks · pass@1: mean of 3 attempt scores/task, then mean across tasks. Exact run dates, builds and row-specific budgets/retries are unresolved. Different agents are system comparisons. Do not merge official Harbor scores with AA scores; 4.0 differs from2.0/2.1.
| Model | Agent / harness | Effort | Score | Reported run tokens | Reported run USD | Input tokens | Output tokens | Time | Attributable completion USD | Actual billed USD | Source |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | Claude Code | max | 66.0% | Source | |||||||
| Claude Opus 5.5 | Claude Code | max | 63.0% | Source | |||||||
| Claude Sonnet 5.5 | Claude Code | xhigh | 58.0% | Source | |||||||
| Claude Fable 5.1 | Claude Code · mixed-model fallback | max | 58.0% | Source | |||||||
| Claude Sonnet 5.5 | Claude Code | high | 42.0% | Source | |||||||
| Claude Sonnet 5.5 | Claude Code | medium | 27.0% | Source | |||||||
| Claude Sonnet 5.5 | Claude Code | low | 25.0% | Source | |||||||
| Claude Haiku 5.5 | Claude Code | xhigh | 24.0% | Source | |||||||
| Claude Haiku 5.5 | Claude Code | max | 30.0% | Source | |||||||
| Claude Haiku 5.5 | Claude Code | high | 13.0% | Source | |||||||
| Claude Haiku 5.5 | Claude Code | medium | 12.0% | Source | |||||||
| Claude Haiku 5.5 | Claude Code | low | 6.0% | Source | |||||||
| GPT-6.1 Sol | Codex | xhigh | 55.0% | Source | |||||||
| GPT-6 Astra | Codex | max | 56.0% | Source | |||||||
| GPT-6.1 Sol | Codex | medium | 52.0% | Source | |||||||
| GPT-6.1 Sol | Codex | high | 50.0% | Source | |||||||
| GPT-6.1 Sol | Codex | max | 53.0% | Source | |||||||
| GPT-6.1 Sol | Codex | low | 49.0% | Source | |||||||
| GPT-6 Luna | Codex | max | 15.0% | Source | |||||||
| Grok 4.7 | Grok Build | xhigh | 33.0% | Source | |||||||
| Muse Wonder Full | Muse Code | max | 59.0% | Source | |||||||
| Muse Wonder | Muse Code | max | 47.0% | Source | |||||||
| Muse Spark 1.3 | Muse Code | max | 32.0% | Source | |||||||
| Muse Spark 1.3 | Muse Code | xhigh | 17.0% | Source | |||||||
| Gemini 3.5 Flash-Lite | Model docs | ||||||||||
| Gemini 3.8 Flash | Model docs | ||||||||||
| Gemini 3.1 Pro preview | Model docs |
Method · captured 2026-10-13; evaluation dates unavailable.
SWE-Atlas-QnA · Artificial Analysis · v1.5
124 tasks · pass@1: mean of 3 attempt scores/task, then mean across tasks. Exact run dates, builds and row-specific budgets/retries are unresolved. Rubric Q&A, not patch coding. Separate Scale and AA cohorts; published pooled efficiency belongs to all303tasks.
| Model | Agent / harness | Effort | Score | Reported run tokens | Reported run USD | Input tokens | Output tokens | Time | Attributable completion USD | Actual billed USD | Source |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | Claude Code | max | 67.0% | Source | |||||||
| Claude Opus 5.5 | Claude Code | max | 66.0% | Source | |||||||
| Claude Sonnet 5.5 | Claude Code | xhigh | 62.0% | Source | |||||||
| Claude Fable 5.1 | Claude Code · mixed-model fallback | max | 65.0% | Source | |||||||
| Claude Sonnet 5.5 | Claude Code | high | 56.0% | Source | |||||||
| Claude Sonnet 5.5 | Claude Code | medium | 45.0% | Source | |||||||
| Claude Sonnet 5.5 | Claude Code | low | 39.0% | Source | |||||||
| Claude Haiku 5.5 | Claude Code | xhigh | 52.0% | Source | |||||||
| Claude Haiku 5.5 | Claude Code | max | 34.0% | Source | |||||||
| Claude Haiku 5.5 | Claude Code | high | 46.0% | Source | |||||||
| Claude Haiku 5.5 | Claude Code | medium | 42.0% | Source | |||||||
| Claude Haiku 5.5 | Claude Code | low | 33.0% | Source | |||||||
| GPT-6.1 Sol | Codex | xhigh | 61.0% | Source | |||||||
| GPT-6 Astra | Codex | max | 62.0% | Source | |||||||
| GPT-6.1 Sol | Codex | medium | 61.0% | Source | |||||||
| GPT-6.1 Sol | Codex | high | 60.0% | Source | |||||||
| GPT-6.1 Sol | Codex | max | 58.0% | Source | |||||||
| GPT-6.1 Sol | Codex | low | 55.0% | Source | |||||||
| GPT-6 Luna | Codex | max | 44.0% | Source | |||||||
| Grok 4.7 | Grok Build | xhigh | 63.0% | Source | |||||||
| Muse Wonder Full | Muse Code | max | 68.0% | Source | |||||||
| Muse Wonder | Muse Code | max | 67.0% | Source | |||||||
| Muse Spark 1.3 | Muse Code | max | 59.0% | Source | |||||||
| Muse Spark 1.3 | Muse Code | xhigh | 55.0% | Source | |||||||
| Gemini 3.5 Flash-Lite | Model docs | ||||||||||
| Gemini 3.8 Flash | Model docs | ||||||||||
| Gemini 3.1 Pro preview | Model docs |
Method · captured 2026-10-13; evaluation dates unavailable.
SWE-Atlas-QnA · Scale · owner excerpt
124 tasks · Task Resolve Rate: every rubric criterion passes. Exact run dates, builds and row-specific budgets/retries are unresolved. Rubric Q&A, not patch coding. Separate Scale and AA cohorts; published pooled efficiency belongs to all303tasks.
| Model | Agent / harness | Effort | Score | Reported run tokens | Reported run USD | Input tokens | Output tokens | Time | Attributable completion USD | Actual billed USD | Source |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Fable 5.1 | Claude Code | xhigh | 59.95% ± 4.85 pp | Source | |||||||
| GPT-6 Astra | Codex | xhigh | 59.14% ± 4.88 pp | Source | |||||||
| Gemini 3.8 Flash | mini-SWE-agent | 47.04% ± 5.08 pp | Source | ||||||||
| GPT-6 Luna | Model docs | ||||||||||
| GPT-6.1 Sol | Model docs | ||||||||||
| Claude Haiku 5.5 | Model docs | ||||||||||
| Claude Sonnet 5.5 | Model docs | ||||||||||
| Claude Opus 5.5 | Model docs | ||||||||||
| Gemini 3.5 Flash-Lite | Model docs | ||||||||||
| Gemini 3.1 Pro preview | Model docs | ||||||||||
| Muse Spark 1.3 | Model docs | ||||||||||
| Muse Wonder | Model docs | ||||||||||
| Muse Wonder Full | Model docs | ||||||||||
| Grok 4.7 | Model docs |
Method · captured 2026-10-13; evaluation dates unavailable.
SWE-bench Pro V2 · HARD-51 · Scale · V2 HARD-51
51 tasks · Resolve Rate. Exact run dates, builds and row-specific budgets/retries are unresolved. Do not mix642Full/51HARD/731v1. Per-row protocol implementation/regrade/repeats require further verification.
| Model | Agent / harness | Effort | Score | Reported run tokens | Reported run USD | Input tokens | Output tokens | Time | Attributable completion USD | Actual billed USD | Source |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Fable 5.1 | Claude Code | high | 92.2% | Source | |||||||
| GPT-6 Astra | Codex | high | 90.2% | Source | |||||||
| Claude Sonnet 5.5 | mini-swe-agent | xhigh | 70.59% | Source | |||||||
| Gemini 3.8 Flash | mini-swe-agent | high | 58.8% | Source | |||||||
| Claude Haiku 5.5 | mini-swe-agent | high | 52.94% ± 13.01 pp | Source | |||||||
| GPT-6 Luna | Model docs | ||||||||||
| GPT-6.1 Sol | Model docs | ||||||||||
| Claude Opus 5.5 | Model docs | ||||||||||
| Gemini 3.5 Flash-Lite | Model docs | ||||||||||
| Gemini 3.1 Pro preview | Model docs | ||||||||||
| Muse Spark 1.3 | Model docs | ||||||||||
| Muse Wonder | Model docs | ||||||||||
| Muse Wonder Full | Model docs | ||||||||||
| Grok 4.7 | Model docs |
Method · captured 2026-10-13; evaluation dates unavailable.
Additional benchmark History · 44 observations
Exact older labels and settings stay here. Unknown latest status for other families remains unverified; these rows are not proxies for current versions.
| Original model label | Benchmark | Evaluation source | Agent / harness | Effort | Score | Reported run USD | Source |
|---|---|---|---|---|---|---|---|
| Opus 5 | Terminal-Bench 4.0.0 | Harbor · official board | Claude Code | xhigh | 53.9% ± 3.2 pp | $6.1k | Source |
| Opus 5 | Terminal-Bench 4.0.0 | Harbor · official board | Claude Code | max | 51.8% ± 3.4 pp | $6.0k | Source |
| Opus 5 | Terminal-Bench 4.0.0 | Harbor · official board | Claude Code | high | 50.3% ± 3.7 pp | $4.7k | Source |
| GPT-6 Sol | Terminal-Bench 4.0.0 | Harbor · official board | Codex | max | 49.4% ± 3.2 pp | $1.5k | Source |
| Opus 5 | Terminal-Bench 4.0.0 | Harbor · official board | Claude Code | medium | 44.8% ± 3.8 pp | $3.2k | Source |
| Fable 5 | Terminal-Bench 4.0.0 | Harbor · official board | Claude Code | max | 44.5% ± 3.8 pp | $7.3k | Source |
| GLM-5.3 | Terminal-Bench 4.0.0 | Harbor · official board | Claude Code | max | 41.8% ± 3.2 pp | $2.7k | Source |
| GPT-5.6 Sol | Terminal-Bench 4.0.0 | Harbor · official board | Codex | max | 37.3% ± 3.8 pp | $2.5k | Source |
| GLM-5.3-Flash | Terminal-Bench 4.0.0 | Harbor · official board | Claude Code | none | 35.8% ± 3.5 pp | $2.9k | Source |
| Opus 5 | Terminal-Bench 4.0.0 | Harbor · official board | Claude Code | low | 34.8% ± 3.9 pp | $2.4k | Source |
| Qwen3.8-Max-0902 | Terminal-Bench 4.0.0 | Harbor · official board | Claude Code | max | 27.0% ± 3.8 pp | $4.6k | Source |
| Opus 4.8 | Terminal-Bench 4.0.0 | Harbor · official board | Claude Code | max | 23.6% ± 3.6 pp | $6.5k | Source |
| GPT-5.6 Terra | Terminal-Bench 4.0.0 | Harbor · official board | Codex | max | 21.5% ± 3.3 pp | $1.7k | Source |
| Grok 4.6 | Terminal-Bench 4.0.0 | Harbor · official board | Grok Build | high | 20.3% ± 3.1 pp | $3.6k | Source |
| GPT-5.6 Luna | Terminal-Bench 4.0.0 | Harbor · official board | Codex | max | 17.3% ± 2.8 pp | $346.67 | Source |
| Grok 4.5 | Terminal-Bench 4.0.0 | Harbor · official board | Grok Build | high | 12.4% ± 2.6 pp | $2.1k | Source |
| Sonnet 5 | Terminal-Bench 4.0.0 | Harbor · official board | Claude Code | max | 12.4% ± 3.1 pp | $9.6k | Source |
| Gemini 3.7 Flash | Terminal-Bench 4.0.0 | Harbor · official board | mini-SWE-agent | high | 11.2% ± 2.4 pp | $1.3k | Source |
| Opus 5 | Terminal-Bench 4.0.0 | Artificial Analysis · v1.5 | Claude Code | max | 55.0% | Source | |
| Opus 5 | SWE-Atlas-QnA | Artificial Analysis · v1.5 | Claude Code | max | 62.0% | Source | |
| Step 5 Preview | Terminal-Bench 4.0.0 | Artificial Analysis · v1.5 | Claude Code | unspecified | 33.0% | Source | |
| Step 5 Preview | SWE-Atlas-QnA | Artificial Analysis · v1.5 | Claude Code | unspecified | 58.0% | Source | |
| Qwen3.8 Max | Terminal-Bench 4.0.0 | Artificial Analysis · v1.5 | Claude Code | unspecified | 17.0% | Source | |
| Qwen3.8 Max | SWE-Atlas-QnA | Artificial Analysis · v1.5 | Claude Code | unspecified | 62.0% | Source | |
| GPT-6 Sol | Terminal-Bench 4.0.0 | Artificial Analysis · v1.5 | Codex | max | 43.0% | Source | |
| GPT-6 Sol | SWE-Atlas-QnA | Artificial Analysis · v1.5 | Codex | max | 58.0% | Source | |
| GPT-5.6 Sol | Terminal-Bench 4.0.0 | Artificial Analysis · v1.5 | Codex | max | 37.0% | Source | |
| GPT-5.6 Sol | SWE-Atlas-QnA | Artificial Analysis · v1.5 | Codex | max | 54.0% | Source | |
| GPT-5.6 Luna | Terminal-Bench 4.0.0 | Artificial Analysis · v1.5 | Codex | max | 15.0% | Source | |
| GPT-5.6 Luna | SWE-Atlas-QnA | Artificial Analysis · v1.5 | Codex | max | 49.0% | Source | |
| DeepSeek V4 Pro 0813 | Terminal-Bench 4.0.0 | Artificial Analysis · v1.5 | Codex | max | 10.0% | Source | |
| DeepSeek V4 Pro 0813 | SWE-Atlas-QnA | Artificial Analysis · v1.5 | Codex | max | 62.0% | Source | |
| DeepSeek V4 Flash 0731 | Terminal-Bench 4.0.0 | Artificial Analysis · v1.5 | Codex | max | 11.0% | Source | |
| DeepSeek V4 Flash 0731 | SWE-Atlas-QnA | Artificial Analysis · v1.5 | Codex | max | 51.0% | Source | |
| Grok 4.6 | Terminal-Bench 4.0.0 | Artificial Analysis · v1.5 | Grok Build | xhigh | 18.0% | Source | |
| Grok 4.6 | SWE-Atlas-QnA | Artificial Analysis · v1.5 | Grok Build | xhigh | 58.0% | Source | |
| Opus 5 | SWE-bench Pro V2 · HARD-51 | Scale · V2 HARD-51 | Claude Code | xhigh | 98.0% | Source | |
| Kimi-K3 | SWE-bench Pro V2 · HARD-51 | Scale · V2 HARD-51 | mini-swe-agent | max | 88.2% | Source | |
| Sonnet 5 | SWE-bench Pro V2 · HARD-51 | Scale · V2 HARD-51 | Claude Code | xhigh | 88.2% | Source | |
| GPT-5.6 Terra | SWE-bench Pro V2 · HARD-51 | Scale · V2 HARD-51 | Codex | xhigh | 86.3% | Source | |
| GLM-5.3 | SWE-bench Pro V2 · HARD-51 | Scale · V2 HARD-51 | mini-swe-agent | max | 84.3% | Source | |
| GPT-5.6 Sol | SWE-bench Pro V2 · HARD-51 | Scale · V2 HARD-51 | Codex | xhigh | 82.4% | Source | |
| Inkling | SWE-bench Pro V2 · HARD-51 | Scale · V2 HARD-51 | mini-swe-agent | xhigh | 56.9% | Source | |
| Haiku 4.5 | SWE-bench Pro V2 · HARD-51 | Scale · V2 HARD-51 | Claude Code | xhigh | 25.5% | Source |
AA suite-wide efficiency · 303 pooled tasks
Per-task-attempt averages pool DeepSWE v1.1, Terminal-Bench 4.0 and SWE-Atlas-QnA. These values never provide a cost coordinate for one benchmark. Time covers the active agent process, excluding environment startup and verifier/judge overhead. Token categories, telemetry coverage and pricing-effective dates are unresolved; API charges are estimates, not invoices. Missing Muse charges are unavailable.
| Model / original label | Bucket | Agent / harness | Effort | Reported total tokens / attempt | Reported agent time / attempt | Estimated API USD / attempt | Actual billed USD | Source |
|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | Current | Claude Code | max | 27.7M | 1.5h | 14.19 | Source | |
| Claude Opus 5.5 | Current | Claude Code | max | 15.6M | 1.1h | 13.04 | Source | |
| Claude Sonnet 5.5 | Current | Claude Code | xhigh | 7.1M | 27.0m | 3.33 | Source | |
| Claude Fable 5.1 | Current | Claude Code · mixed-model fallback | max | 5.7M | 34.8m | 12.39 | Source | |
| Opus 5 | History | Claude Code | max | 11.4M | 41.9m | 10.79 | Source | |
| Claude Sonnet 5.5 | Current | Claude Code | high | 2.7M | 12.3m | 1.24 | Source | |
| Step 5 Preview | History | Claude Code | unspecified | 21.7M | 56.4m | 1.66 | Source | |
| Claude Sonnet 5.5 | Current | Claude Code | medium | 1.3M | 8.5m | 0.62 | Source | |
| Qwen3.8 Max | History | Claude Code | unspecified | 8.7M | 1.1h | 3.48 | Source | |
| Claude Sonnet 5.5 | Current | Claude Code | low | 977.3K | 6.3m | 0.48 | Source | |
| Claude Haiku 5.5 | Current | Claude Code | xhigh | 7.1M | 25.9m | 0.61 | Source | |
| Claude Haiku 5.5 | Current | Claude Code | max | 13.8M | 1.5h | 2.58 | Source | |
| Claude Haiku 5.5 | Current | Claude Code | high | 5.1M | 18.4m | 0.36 | Source | |
| Claude Haiku 5.5 | Current | Claude Code | medium | 3.5M | 14.0m | 0.23 | Source | |
| Claude Haiku 5.5 | Current | Claude Code | low | 2.4M | 11.6m | 0.14 | Source | |
| GPT-6.1 Sol | Current | Codex | xhigh | 3.2M | 15.5m | 1.04 | Source | |
| GPT-6 Astra | Current | Codex | max | 3.3M | 29.4m | 7.47 | Source | |
| GPT-6.1 Sol | Current | Codex | medium | 2.3M | 10.9m | 0.7 | Source | |
| GPT-6.1 Sol | Current | Codex | high | 2.8M | 13.3m | 0.89 | Source | |
| GPT-6.1 Sol | Current | Codex | max | 4M | 24.4m | 1.55 | Source | |
| GPT-6.1 Sol | Current | Codex | low | 1.7M | 8.6m | 0.5 | Source | |
| GPT-6 Sol | History | Codex | max | 9.8M | 22.3m | 2.99 | Source | |
| GPT-5.6 Sol | History | Codex | max | 10.2M | 20.6m | 6.35 | Source | |
| GPT-5.6 Luna | History | Codex | max | 14.9M | 23.5m | 0.44 | Source | |
| DeepSeek V4 Pro 0813 | History | Codex | max | 24.2M | 40.3m | 0.24 | Source | |
| GPT-6 Luna | Current | Codex | max | 10.2M | 21.4m | 0.18 | Source | |
| DeepSeek V4 Flash 0731 | History | Codex | max | 15.4M | 18.3m | 0.09 | Source | |
| Grok 4.7 | Current | Grok Build | xhigh | 14.3M | 39.2m | 8.82 | Source | |
| Grok 4.6 | History | Grok Build | xhigh | 5.5M | 19.5m | 3.57 | Source | |
| Muse Wonder Full | Current | Muse Code | max | 15M | 28.8m | Source | ||
| Muse Wonder | Current | Muse Code | max | 15M | 22.4m | 8.05 | Source | |
| Muse Spark 1.3 | Current | Muse Code | max | 16.7M | 18.4m | 3.98 | Source | |
| Muse Spark 1.3 | Current | Muse Code | xhigh | 16.2M | 18.3m | 3.47 | Source |
DeepSWE evidence tables
How to read the tables
Blank cells mean unavailable, not zero or a lower score. A new model does not inherit an older version’s measurements. Exact serving snapshots, run dates, harness builds, token budgets, retries and speed tiers are not established; equal effort names do not establish equal compute.
Latest model names, IDs and availability
| Official name | API ID | Release date | Availability | Source |
|---|---|---|---|---|
| GPT-6 Luna | gpt-6-luna | Documented API model; Work/Codex access depends on rollout/client/workspace. | Official | |
| GPT-6.1 Sol | gpt-6.1-sol | Documented API model; Work/Codex access depends on rollout/client/workspace. | Official | |
| GPT-6 Astra | gpt-6-astra | Documented API model. | Official | |
| Claude Haiku 5.5 | claude-haiku-5-5 | 2026-10-07 | Released on Claude Platform and partners. | Official |
| Claude Sonnet 5.5 | claude-sonnet-5-5 | 2026-09-28 | Released on Claude Platform and partners. | Official |
| Claude Opus 5.5 | claude-opus-5-5 | 2026-09-22 | Released on Claude Platform and partners. | Official |
| Claude Fable 5.1 | claude-fable-5-1 | Current documented model for demanding reasoning. | Official | |
| Gemini 3.5 Flash-Lite | gemini-3.5-flash-lite | Stable text-output API model; excludes TTS/image variants. | Official | |
| Gemini 3.8 Flash | gemini-3.8-flash | Stable text-output API model; excludes TTS/image/live variants. | Official | |
| Gemini 3.1 Pro preview | gemini-3.1-pro-preview | Preview. Board label gemini-3.1-pro does not establish this exact endpoint/snapshot. | Official | |
| Muse Spark 1.3 | 2026-09-02 | Meta Model API and Muse Code; exact API ID unverified. | Official | |
| Muse Wonder | 2026-10-12 | Meta Model API and Muse Code; exact API ID unverified. | Official | |
| Muse Wonder Full | 2026-10-12 | Muse Code subscribers; public API access/ID unverified. | Official | |
| Grok 4.7 | grok-4.7 | 2026-09-21 | Documented API flagship; preserve xAI/Grok user label. | Official |
Meta API IDs are unverified. The board’s Gemini 3.1 Pro label is not matched to the exact preview endpoint. Restricted Gemini 4 Argon is outside the current graph.
Benchmark score
| Latest model | Effort | Score (%) | Interval (± pp) | Source |
|---|---|---|---|---|
| GPT-6 Luna | DeepSWE | |||
| GPT-6.1 Sol | DeepSWE | |||
| GPT-6 Astra | xhigh | 74 | 3 | DeepSWE |
| GPT-6 Astra | high | 73 | 3 | DeepSWE |
| GPT-6 Astra | max | 73 | 1 | DeepSWE |
| GPT-6 Astra | medium | 73 | 3 | DeepSWE |
| GPT-6 Astra | low | 67 | 1 | DeepSWE |
| Claude Haiku 5.5 | DeepSWE | |||
| Claude Sonnet 5.5 | DeepSWE | |||
| Claude Opus 5.5 | DeepSWE | |||
| Claude Fable 5.1 | DeepSWE | |||
| Gemini 3.5 Flash-Lite | DeepSWE | |||
| Gemini 3.8 Flash | high | 74 | 1 | DeepSWE |
| Gemini 3.8 Flash | medium | 71 | 2 | DeepSWE |
| Gemini 3.1 Pro preview | DeepSWE | |||
| Muse Spark 1.3 | DeepSWE | |||
| Muse Wonder | DeepSWE | |||
| Muse Wonder Full | DeepSWE | |||
| Grok 4.7 | DeepSWE |
Tokens
The leaderboard exposes rounded average output tokens. Actual input, cache, output and reasoning counters are unavailable at this aggregate scope; no total is reconstructed.
| Latest model | Effort | Input | Uncached input | Cache read | Cache write | Output | Reasoning | Avg output (rounded) | Source |
|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Luna | DeepSWE | ||||||||
| GPT-6.1 Sol | DeepSWE | ||||||||
| GPT-6 Astra | xhigh | 30k | DeepSWE | ||||||
| GPT-6 Astra | high | 27k | DeepSWE | ||||||
| GPT-6 Astra | max | 61k | DeepSWE | ||||||
| GPT-6 Astra | medium | 20k | DeepSWE | ||||||
| GPT-6 Astra | low | 11k | DeepSWE | ||||||
| Claude Haiku 5.5 | DeepSWE | ||||||||
| Claude Sonnet 5.5 | DeepSWE | ||||||||
| Claude Opus 5.5 | DeepSWE | ||||||||
| Claude Fable 5.1 | DeepSWE | ||||||||
| Gemini 3.5 Flash-Lite | DeepSWE | ||||||||
| Gemini 3.8 Flash | high | 143k | DeepSWE | ||||||
| Gemini 3.8 Flash | medium | 125k | DeepSWE | ||||||
| Gemini 3.1 Pro preview | DeepSWE | ||||||||
| Muse Spark 1.3 | DeepSWE | ||||||||
| Muse Wonder | DeepSWE | ||||||||
| Muse Wonder Full | DeepSWE | ||||||||
| Grok 4.7 | DeepSWE |
Time
| Latest model | Effort | Duration | Unit | Measure | Source |
|---|---|---|---|---|---|
| GPT-6 Luna | DeepSWE | ||||
| GPT-6.1 Sol | DeepSWE | ||||
| GPT-6 Astra | xhigh | DeepSWE | |||
| GPT-6 Astra | high | DeepSWE | |||
| GPT-6 Astra | max | DeepSWE | |||
| GPT-6 Astra | medium | DeepSWE | |||
| GPT-6 Astra | low | DeepSWE | |||
| Claude Haiku 5.5 | DeepSWE | ||||
| Claude Sonnet 5.5 | DeepSWE | ||||
| Claude Opus 5.5 | DeepSWE | ||||
| Claude Fable 5.1 | DeepSWE | ||||
| Gemini 3.5 Flash-Lite | DeepSWE | ||||
| Gemini 3.8 Flash | high | DeepSWE | |||
| Gemini 3.8 Flash | medium | DeepSWE | |||
| Gemini 3.1 Pro preview | DeepSWE | ||||
| Muse Spark 1.3 | DeepSWE | ||||
| Muse Wonder | DeepSWE | ||||
| Muse Wonder Full | DeepSWE | ||||
| Grok 4.7 | DeepSWE |
Steps
| Latest model | Effort | Average steps | Source |
|---|---|---|---|
| GPT-6 Luna | DeepSWE | ||
| GPT-6.1 Sol | DeepSWE | ||
| GPT-6 Astra | xhigh | 29 | DeepSWE |
| GPT-6 Astra | high | 27 | DeepSWE |
| GPT-6 Astra | max | 28 | DeepSWE |
| GPT-6 Astra | medium | 26 | DeepSWE |
| GPT-6 Astra | low | 20 | DeepSWE |
| Claude Haiku 5.5 | DeepSWE | ||
| Claude Sonnet 5.5 | DeepSWE | ||
| Claude Opus 5.5 | DeepSWE | ||
| Claude Fable 5.1 | DeepSWE | ||
| Gemini 3.5 Flash-Lite | DeepSWE | ||
| Gemini 3.8 Flash | high | 166 | DeepSWE |
| Gemini 3.8 Flash | medium | 147 | DeepSWE |
| Gemini 3.1 Pro preview | DeepSWE | ||
| Muse Spark 1.3 | DeepSWE | ||
| Muse Wonder | DeepSWE | ||
| Muse Wonder Full | DeepSWE | ||
| Grok 4.7 | DeepSWE |
Cost
Astra uses expected launch-rate estimates dated September 3. Other reported cost calculations are not fully verified. Billed totals, complete benchmark cost and retry-inclusive cost per successful completion remain blank. Pricing and reporting changes
| Latest model | Effort | Reported avg / attempt | Billed cost | Benchmark total | Retry-inclusive / success | Basis / effective date | Source |
|---|---|---|---|---|---|---|---|
| GPT-6 Luna | DeepSWE | ||||||
| GPT-6.1 Sol | DeepSWE | ||||||
| GPT-6 Astra | xhigh | $4.43 | Expected launch-rate estimate · 2026-09-03 | DeepSWE | |||
| GPT-6 Astra | high | $3.92 | Expected launch-rate estimate · 2026-09-03 | DeepSWE | |||
| GPT-6 Astra | max | $7.50 | Expected launch-rate estimate · 2026-09-03 | DeepSWE | |||
| GPT-6 Astra | medium | $3.08 | Expected launch-rate estimate · 2026-09-03 | DeepSWE | |||
| GPT-6 Astra | low | $1.60 | Expected launch-rate estimate · 2026-09-03 | DeepSWE | |||
| Claude Haiku 5.5 | DeepSWE | ||||||
| Claude Sonnet 5.5 | DeepSWE | ||||||
| Claude Opus 5.5 | DeepSWE | ||||||
| Claude Fable 5.1 | DeepSWE | ||||||
| Gemini 3.5 Flash-Lite | DeepSWE | ||||||
| Gemini 3.8 Flash | high | $2.36 | Source-reported; pricing basis unverified | DeepSWE | |||
| Gemini 3.8 Flash | medium | $1.97 | Source-reported; pricing basis unverified | DeepSWE | |||
| Gemini 3.1 Pro preview | DeepSWE | ||||||
| Muse Spark 1.3 | DeepSWE | ||||||
| Muse Wonder | DeepSWE | ||||||
| Muse Wonder Full | DeepSWE | ||||||
| Grok 4.7 | DeepSWE |
One task, two trials
happy-dom-abort-pending-body-reads · DeepSWE v1.1. These are individual inspected trials, separate from leaderboard averages. Both report a pass; equal “high” settings do not establish equivalent budgets.
| Model | Effort | Outcome | Fail → pass tests | Pass → pass tests | Run started (UTC) | Source |
|---|---|---|---|---|---|---|
| GPT-6 Astra | high | pass | 2026-08-31T23:15:39Z | Trial | ||
| Gemini 3.8 Flash | high | pass | 14/14 | 165/165 | 2026-08-29T08:25:20.393198Z | Trial |
| Model | Effort | Input incl. cache | Uncached input | Cache read | Cache write | Output | Reasoning | Source |
|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra | high | 2,645,526 | 987 | 2,528,884 | 115,655 | 28,879 | 11,206 | Trial |
| Gemini 3.8 Flash | high | 47,636,992 | 47,374,336 | 221,856 | Trial |
Input includes cached input. Output/reasoning overlap is unresolved; these columns must not be added into a fabricated total. Astra’s trajectory/verifier/patch tabs are disabled; Gemini’s raw trajectory was not audited.
| Model | Effort | Trial (min) | Agent (min) | Agent steps | Source |
|---|---|---|---|---|---|
| GPT-6 Astra | high | 18.3 | 18.3 | 37 | Trial |
| Gemini 3.8 Flash | high | 19.4 | 18.5 | 247 | Trial |
| Model | Effort | Reported cost | Billed cost | Completion total | Basis / effective date | Source |
|---|---|---|---|---|---|---|
| GPT-6 Astra | high | $5.43 | Expected launch-rate estimate · 2026-09-03 | Trial | ||
| Gemini 3.8 Flash | high | $4.58 | Source-reported; pricing basis unverified | Trial |
Task and evaluation conditions
Base commit: 82a0888cb2c87a6123e05424b528f8e8c9b3e426. Harness: mini-swe-agent / Pier / Modal. Declared agent timeout: 5,400 seconds; verifier timeout: 1,800 seconds. Enforced token budget, retry policy, exact harness build and speed tier are unknown.
Trial IDs: happy-dom-abort-pending-body-rea__5merruF and happy-dom-abort-pending-body-rea__8DdxqX3. Astra metrics come from the verified OpenAI handoff; Gemini metrics come from the Vertex AI trial summary. Reported costs are not actual invoices or complete completion costs.
History
Older model versions and unmatched source records are retained here. The Datacurve archive uses the same September 22 source update and October 12 capture as the current tables.
Older versions · 43 configurations
| Exact source label | Effort | Score (%) | Interval (± pp) | Avg output (rounded) | Avg steps | Avg USD / attempt | Basis / effective date | Source |
|---|---|---|---|---|---|---|---|---|
| claude-opus-5 | max | 74 | 4 | 118k | 99 | $11.84 | Source-reported; pricing basis unverified | DeepSWE |
| claude-opus-5 | xhigh | 73 | 3 | 92k | 89 | $9.07 | Source-reported; pricing basis unverified | DeepSWE |
| claude-opus-5 | high | 73 | 2 | 64k | 73 | $6.08 | Source-reported; pricing basis unverified | DeepSWE |
| gpt-5.6-sol | max | 73 | 3 | 60k | 61 | $6.46 | Source-reported; pricing basis unverified | DeepSWE |
| gpt-5.6-sol | xhigh | 71 | 1 | 41k | 44 | $3.60 | Source-reported; pricing basis unverified | DeepSWE |
| claude-fable-5 | xhigh | 70 | 3 | 80k | 68 | $13.41 | Source-reported; pricing basis unverified | DeepSWE |
| claude-fable-5 | max | 70 | 4 | 119k | 88 | $21.63 | Source-reported; pricing basis unverified | DeepSWE |
| gpt-5.6-sol | high | 69 | 1 | 28k | 37 | $2.66 | Source-reported; pricing basis unverified | DeepSWE |
| claude-opus-5 | medium | 69 | 1 | 37k | 52 | $3.29 | Source-reported; pricing basis unverified | DeepSWE |
| claude-fable-5 | high | 69 | 1 | 57k | 59 | $9.18 | Source-reported; pricing basis unverified | DeepSWE |
| grok-4.6 | medium | 67 | 2 | 50k | 70 | $3.45 | Source-reported; pricing basis unverified | DeepSWE |
| gpt-5.6-luna | max | 67 | 4 | 73k | 102 | $0.61 | Source-reported; pricing basis unverified | DeepSWE |
| grok-4.6 | xhigh | 67 | 2 | 71k | 87 | $5.50 | Source-reported; pricing basis unverified | DeepSWE |
| gemini-3.7-flash | medium | 65 | 3 | 94k | 117 | $2.03 | Source-reported; pricing basis unverified | DeepSWE |
| claude-fable-5 | medium | 65 | 4 | 40k | 48 | $6.09 | Source-reported; pricing basis unverified | DeepSWE |
| gemini-3.7-flash | high | 65 | 2 | 107k | 125 | $2.18 | Source-reported; pricing basis unverified | DeepSWE |
| grok-4.6 | high | 65 | 2 | 61k | 79 | $4.38 | Source-reported; pricing basis unverified | DeepSWE |
| gpt-5.6-sol | medium | 61 | 2 | 18k | 31 | $1.42 | Source-reported; pricing basis unverified | DeepSWE |
| claude-fable-5 | low | 60 | 3 | 25k | 38 | $3.76 | Source-reported; pricing basis unverified | DeepSWE |
| claude-opus-4.8 | max | 59 | 2 | 135k | 120 | $13.22 | Source-reported; pricing basis unverified | DeepSWE |
| claude-opus-5 | low | 58 | 2 | 20k | 36 | $1.66 | Source-reported; pricing basis unverified | DeepSWE |
| gpt-5.6-luna | xhigh | 57 | 2 | 45k | 71 | $0.31 | Source-reported; pricing basis unverified | DeepSWE |
| muse-spark-1.2 | xhigh | 55 | 2 | 99k | 101 | $3.70 | Source-reported; pricing basis unverified | DeepSWE |
| claude-opus-4.8 | xhigh | 54 | 4 | 86k | 95 | $8.01 | Source-reported; pricing basis unverified | DeepSWE |
| claude-sonnet-5 | max | 54 | 4 | 214k | 268 | $26.40 | Source-reported; pricing basis unverified | DeepSWE |
| gemini-3.7-flash | low | 54 | 3 | 73k | 130 | $1.83 | Source-reported; pricing basis unverified | DeepSWE |
| grok-4.5 | high | 54 | 2 | 36k | 61 | $2.42 | Source-reported; pricing basis unverified | DeepSWE |
| muse-spark-1.1 | xhigh | 53 | 3 | 74k | 96 | $2.36 | Source-reported; pricing basis unverified | DeepSWE |
| claude-opus-4.8 | high | 52 | 5 | 50k | 73 | $4.28 | Source-reported; pricing basis unverified | DeepSWE |
| claude-sonnet-5 | xhigh | 50 | 3 | 121k | 186 | $11.89 | Source-reported; pricing basis unverified | DeepSWE |
| claude-opus-4.8 | medium | 49 | 2 | 41k | 66 | $3.44 | Source-reported; pricing basis unverified | DeepSWE |
| claude-sonnet-5 | high | 48 | 5 | 87k | 147 | $7.43 | Source-reported; pricing basis unverified | DeepSWE |
| gemini-3.6-flash | high | 47 | 4 | 96k | 117 | $2.21 | Source-reported; pricing basis unverified | DeepSWE |
| gpt-5.6-sol | low | 45 | 2 | 11k | 23 | $0.82 | Source-reported; pricing basis unverified | DeepSWE |
| gpt-5.6-luna | high | 44 | 3 | 26k | 49 | $0.16 | Source-reported; pricing basis unverified | DeepSWE |
| grok-4.6 | low | 42 | 2 | 16k | 44 | $1.04 | Source-reported; pricing basis unverified | DeepSWE |
| claude-opus-4.8 | low | 41 | 1 | 29k | 54 | $2.29 | Source-reported; pricing basis unverified | DeepSWE |
| claude-sonnet-5 | medium | 40 | 3 | 57k | 108 | $4.08 | Source-reported; pricing basis unverified | DeepSWE |
| gemini-3.5-flash | high | 36 | 4 | 76k | 105 | $3.45 | Source-reported; pricing basis unverified | DeepSWE |
| claude-sonnet-5 | low | 31 | 1 | 36k | 77 | $2.19 | Source-reported; pricing basis unverified | DeepSWE |
| claude-sonnet-4.6 | high | 30 | 4 | 76k | 134 | $5.52 | Source-reported; pricing basis unverified | DeepSWE |
| gpt-5.6-luna | medium | 11 | 1 | 8.2k | 24 | $0.04 | Source-reported; pricing basis unverified | DeepSWE |
| gpt-5.6-luna | low | 2 | 1 | 3.1k | 12 | $0.01 | Source-reported; pricing basis unverified | DeepSWE |
Unmatched source records · 20 configurations
These include families whose latest version was not checked in this inventory and the unresolved Gemini 3.1 Pro endpoint match. They are not automatically obsolete.
| Exact source label | Effort | Score (%) | Interval (± pp) | Avg output (rounded) | Avg steps | Avg USD / attempt | Basis / effective date | Source |
|---|---|---|---|---|---|---|---|---|
| gpt-5.6-terra | max | 70 | 3 | 72k | 76 | $3.96 | Source-reported; pricing basis unverified | DeepSWE |
| glm-5.3 | max | 69 | 3 | 80k | 124 | $3.99 | Source-reported; pricing basis unverified | DeepSWE |
| kimi-k3 | max | 69 | 5 | 81k | 98 | $4.65 | Source-reported; pricing basis unverified | DeepSWE |
| gpt-5.5 | xhigh | 67 | 6 | 46k | 82 | $7.23 | Source-reported; pricing basis unverified | DeepSWE |
| gpt-5.5 | high | 64 | 3 | 31k | 62 | $5.10 | Source-reported; pricing basis unverified | DeepSWE |
| glm-5.3-flash | max | 63 | 4 | 73k | 123 | $0.24 | Source-reported; pricing basis unverified | DeepSWE |
| deepseek-v4-pro | max | 63 | 6 | 106k | 155 | $1.67 | Source-reported; pricing basis unverified | DeepSWE |
| gpt-5.6-terra | xhigh | 60 | 2 | 40k | 43 | $1.70 | Source-reported; pricing basis unverified | DeepSWE |
| qwen3.8-max | xhigh | 57 | 3 | 95k | 111 | $3.73 | Source-reported; pricing basis unverified | DeepSWE |
| gpt-5.5 | medium | 54 | 3 | 20k | 46 | $2.75 | Source-reported; pricing basis unverified | DeepSWE |
| gpt-5.6-terra | high | 54 | 4 | 22k | 34 | $0.91 | Source-reported; pricing basis unverified | DeepSWE |
| deepseek-v4-flash | max | 53 | 4 | 108k | 153 | $0.46 | Source-reported; pricing basis unverified | DeepSWE |
| gpt-5.4 | xhigh | 52 | 2 | 71k | 70 | $5.65 | Source-reported; pricing basis unverified | DeepSWE |
| glm-5.2 | max | 44 | 2 | 78k | 129 | $3.92 | Source-reported; pricing basis unverified | DeepSWE |
| glm-5.2 | high | 36 | 5 | 54k | 122 | $2.84 | Source-reported; pricing basis unverified | DeepSWE |
| gpt-5.6-terra | medium | 35 | 3 | 12k | 25 | $0.47 | Source-reported; pricing basis unverified | DeepSWE |
| kimi-k2.7-code | 31 | 1 | 59k | 149 | $2.82 | Source-reported; pricing basis unverified | DeepSWE | |
| gpt-5.5 | low | 27 | 2 | 9.4k | 28 | $1.20 | Source-reported; pricing basis unverified | DeepSWE |
| gpt-5.6-terra | low | 24 | 1 | 8.6k | 21 | $0.34 | Source-reported; pricing basis unverified | DeepSWE |
| gemini-3.1-pro | high | 12 | 1 | 28k | 76 | $2.14 | Source-reported; pricing basis unverified | DeepSWE |
Original OpenAI launch data · 15 configurations
Separate September 22 / 29 launch charts, checked October 1. These precise launch values are not reassigned to the current rounded Datacurve rows or used to fill current blank cells.
| Model | Effort | Score (%) | USD / attempted task | Source |
|---|---|---|---|---|
| GPT-6.1 Sol | low | 64.38 | $0.1714 | OpenAI launch |
| GPT-6.1 Sol | medium | 73.01 | $0.4196 | OpenAI launch |
| GPT-6.1 Sol | high | 75.22 | $0.6461 | OpenAI launch |
| GPT-6.1 Sol | xhigh | 71.90 | $0.7886 | OpenAI launch |
| GPT-6.1 Sol | max | 71.90 | $1.5711 | OpenAI launch |
| GPT-6 Luna | low | 2.43 | $0.0057 | OpenAI launch |
| GPT-6 Luna | medium | 44.47 | $0.0518 | OpenAI launch |
| GPT-6 Luna | high | 59.29 | $0.0838 | OpenAI launch |
| GPT-6 Luna | xhigh | 61.28 | $0.1096 | OpenAI launch |
| GPT-6 Luna | max | 66.59 | $0.2169 | OpenAI launch |
| GPT-6 Astra | low | 67.04 | $1.5952 | OpenAI launch |
| GPT-6 Astra | medium | 72.79 | $3.0755 | OpenAI launch |
| GPT-6 Astra | high | 73.23 | $3.9237 | OpenAI launch |
| GPT-6 Astra | xhigh | 74.12 | $4.4291 | OpenAI launch |
| GPT-6 Astra | max | 73.23 | $7.4978 | OpenAI launch |
Full original dataset and dated history preserved: October 1 snapshot · Earlier snapshot
Latest measured models
Sources & method
DeepSWE v1.1 · PASS@1 · 113 tasks · mini-swe-agent. Captured October 12, 2026; leaderboard updated September 22. Latest model names verified October 12.
Starting picks are the cheapest reported configuration within 10 or 3 score points of that cohort’s best. Expected-rate estimates and other reported averages are separate cohorts. Exact run snapshots, budgets and retries are unresolved; these are source-specific benchmark heuristics.
Costs are reported averages or estimates, not invoices. Missing latest measurements are blank in the tables and have no plotted proxy. Leaderboard · Methodology · Changelog