Skip to content
Model Menu
Models
Compare
Leaderboard
Benchmarks
⌘K
Top
Chat
Coding
Agents
Writing
Vision
Image
Video
Speech
Embeddings
Decision
Music
Filters
1
All AI models
All models
2292
Chat & Knowledge
558
Coding
381
Agents & Tool Use
541
Writing & Language
360
Vision & Documents
395
Image Generation & Editing
183
Video Generation
148
Speech & Voice
358
Embeddings & Retrieval
671
Decision models
118
Music & Sound
29
AA Intelligence Index 4.3.2
Arena Text
Humanity's Last Exam
Terminal-Bench 4.0
GDPval-AA 2.1
GPQA Diamond
All benchmarks
196
All benchmarks
196
×
All
196
Overall
1
Knowledge
27
Reasoning
9
Math
56
Science
9
Coding
26
Agents
28
Multimodal
13
Writing
3
Health & law
7
Code review
1
Documents
4
Other
12
Benchmark
By
Leader
Models
Overall
1
AA Intelligence Index
4.3.2
Artificial Analysis combines tests of knowledge, reasoning, coding and tool use into one score
AA
Claude Opus 5.5
57.6
337
Knowledge
27
Arena Text
Human pairwise preferences in the stated text vote pool
Arena
Gemini 4 Argon
1,524
196
+16 versions & slices
creative writing
Human pairwise preferences in the stated text vote pool
Arena
Claude Opus 5.5
1,519
195
English
Human pairwise preferences in the stated text vote pool
Arena
Gemini 4 Argon
1,538
188
instruction following
Human pairwise preferences in the stated text vote pool
Arena
Claude Opus 5.5
1,536
188
non-English
Human pairwise preferences in the stated text vote pool
Arena
Gemini 4 Argon
1,522
188
overall · full cohort
Human pairwise preferences in the stated text vote pool
Arena
Gemini 4 Argon
1,532
188
creative writing · full cohort
Human pairwise preferences in the stated text vote pool
Arena
Claude Opus 5.5
1,534
187
Russian
Human pairwise preferences in the stated text vote pool
Arena
Gemini 4 Argon
1,534
186
Chinese
Human pairwise preferences in the stated text vote pool
Arena
Gemini 4 Argon
1,608
181
Spanish
Human pairwise preferences in the stated text vote pool
Arena
Claude Fable 5.1
1,521
155
German
Human pairwise preferences in the stated text vote pool
Arena
Claude Opus 5
1,526
154
French
Human pairwise preferences in the stated text vote pool
Arena
Claude Fable 5.1
1,527
149
Korean
Human pairwise preferences in the stated text vote pool
Arena
Claude Fable 5.1
1,521
144
Japanese
Human pairwise preferences in the stated text vote pool
Arena
Claude Fable 5.1
1,560
138
Polish
Human pairwise preferences in the stated text vote pool
Arena
Claude Fable 5.1
1,524
133
creative writing · Gemini 1.5 Flash (001) / Gemini 1.5 Pro (001) / Mistral Large 2402
Arena Text · creative writing · Arena creative; source adjustment/filter settings · historical run: Human pairwise preferences in the stated text vote pool
Arena
Mistral Large
1,211
1
overall · Gemini 1.5 Flash (001) / Gemini 1.5 Pro (001) / Mistral Large 2402
Human pairwise preferences in the stated text vote pool
Arena
Mistral Large
1,243
1
IFBench (instructions)
Checks whether the model follows precise formatting and content instructions
AA
Grok 4.3
83.3%
248
AA Omniscience
Accuracy
Measures correct answers on the AA Omniscience factual knowledge test
AA
Claude Fable 5.1
67.2%
321
+1 versions & slices
Hallucination Rate
Measures how often attempted answers are incorrect on the AA Omniscience factual knowledge test
AA
O
MiniCPM5 1B
0.9%
321
AA Omniscience Index
Scores factual knowledge while penalizing incorrect answers to encourage admitting uncertainty
AA
Claude Opus 5.5
46.4
321
SimpleQA Verified
v1
Accuracy on short factual questions with verified answers, answered without web access
Gemini 3.1 Pro Preview
77.5%
83
+1 versions & slices
v1 · October 2026 run
SimpleQA Verified v1 · Kaggle SimpleQA Verified v1; generation settings not stated · historical run: Accuracy on short factual questions with verified answers, answered without web access
Gemma 4 31B
10.7%
1
Mystery Game Puzzles
Models choose the best move in puzzle positions from an undisclosed game
Epoch
GPT-6 Astra
84%
64
+1 versions & slices
August 2026 run
Models choose the best move in puzzle positions from an undisclosed game
Epoch
Gemini 3.1 Pro Preview
29%
1
EQ-Bench
3
Judge-rated responses to social and emotionally nuanced scenarios, scored through the source pairwise ranking
EQ-Bench
Claude Fable 5
2,050
50
+1 versions & slices
4
Active emotional and social intelligence in multi-turn conversations with 120 simulated personas
EQ-Bench
Kimi K3
1,339
1
Reasoning
9
Chess Puzzles
Models choose the best next move from a novel chess position
Epoch
GPT-6 Astra
74%
68
ARC-AGI-2
semi-private
Models infer transformations from examples and solve unseen grid puzzles; the evaluation split is explicit
ARC
GPT-6 Astra
95%
66
+3 versions & slices
public evaluation
Models infer transformations from examples and solve unseen grid puzzles; the evaluation split is explicit
ARC
Claude Fable 5.1
99.2%
63
public evaluation · chain of thought
Models infer transformations from examples and solve unseen grid puzzles; the evaluation split is explicit
ARC
R1 0528
0.3%
1
semi-private · chain of thought
Models infer transformations from examples and solve unseen grid puzzles; the evaluation split is explicit
ARC
R1 0528
1.1%
1
ARC-AGI-1
semi-private
Models infer transformations from examples and solve unseen grid puzzles; the evaluation split is explicit
ARC
Claude Fable 5
98.5%
64
+2 versions & slices
public evaluation
Models infer transformations from examples and solve unseen grid puzzles; the evaluation split is explicit
ARC
Gemini 4 Argon
99.5%
61
semi-private · chain of thought
Models infer transformations from examples and solve unseen grid puzzles; the evaluation split is explicit
ARC
R1 0528
21.2%
1
ARC-AGI-3
semi-private
Measures interactive game solving through relative human action efficiency
ARC
GPT-6 Astra
62.7%
18
Math
53
FrontierMath Tiers 1–3 (v2)
Models solve expert-written advanced mathematics problems on the named tier and edition
Epoch
GPT-6 Astra
93.7%
69
FrontierMath Tier 4 (v2)
Models solve expert-written advanced mathematics problems on the named tier and edition
Epoch
GPT-6.1 Sol
100%
54
AA Analyst Agent
Data-analyst tasks: the model explores data with tools and reports findings
AA
Claude Fable 5.1
57.5%
43
FrontierMath
Models solve expert-written advanced mathematics problems on the named tier and edition
Epoch
GPT-5.4 Pro (2026-03-05)
50%
40
MATH Level
5
Models solve competition mathematics problems from the hardest MATH difficulty level
Epoch
GPT-5 (2025-08-07)
98.1%
35
MathArena
Apex
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
Claude Opus 4.8
81.3%
34
+45 versions & slices
AIME 2025
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.2
100%
30
Apex Shortlist
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.5
98.4%
30
HMMT February 2025
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.2
100%
30
AIME 2026
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
Claude Opus 4.8
100%
24
HMMT February 2026
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.5
98.5%
24
ArXivMath · June 2026
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-6 Astra
94.4%
24
BrokenArXiv · June 2026
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
Claude Opus 5.5
100%
24
BRUMO 2025
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
Gemini 3 Flash Preview
100%
24
SMT 2025
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.2
96%
24
CMIMC 2025
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.2
95.6%
22
ArXivMath · February 2026
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.4 Pro
75.8%
20
ArXivMath · May 2026
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-6 Astra
95%
19
ArXivMath · January 2026
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.4
76.1%
18
BrokenArXiv · May 2026
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
Claude Opus 5.5
95%
18
ArXivMath · April 2026
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-6 Astra
90.8%
17
Kangaroo 2025 1-2
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.5
95.8%
17
Kangaroo 2025 11-12
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
Claude Opus 4.8
100%
17
Kangaroo 2025 3-4
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.5
89.6%
17
Kangaroo 2025 5-6
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.5
90%
17
Kangaroo 2025 7-8
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
Claude Opus 4.8
96.7%
16
Kangaroo 2025 9-10
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
Gemini 3.1 Pro Preview
100%
16
BrokenArXiv · April 2026
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-6 Astra
97.5%
15
BrokenArXiv · February 2026
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.5
68.2%
15
ArXivMath · March 2026
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.5
77.5%
14
HMMT November 2025
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.2
99.2%
14
ArXivMath · December 2025
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
Gemini 3.1 Pro Preview
66.2%
13
BrokenArXiv · March 2026
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.5
73.7%
13
ArXivMath · August 2026
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-6.1 Sol
96.5%
12
BrokenArXiv · August 2026
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-6.1 Sol
86.3%
12
Project Euler
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
Gemini 3.1 Pro Preview
89%
11
ArXivLean · June 2026
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-6 Astra
65.2%
9
USAMO 2026
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.5
98.2%
8
ArXivMath · September 2026
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-6.1 Sol
91.5%
7
BrokenArXiv · September 2026
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-6.1 Sol
82.7%
7
ArXivLean · March 2026
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.4
17.1%
5
ArXivLean · September 2026
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
Claude Opus 5.5
47.2%
5
IMO 2025
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5
38.1%
4
Putnam 2025
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5 Pro
77.5%
3
USAMO 2025
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
R1 0528
30.1%
3
Miklós Schweitzer 2025
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5 Pro
90%
1
AIME 2025 · DeepSeek R1 (original) / o3-mini · low effort
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
R1 0528
70%
1
BRUMO 2025 · DeepSeek R1 (original)
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
R1 0528
80.8%
1
HMMT February 2025 · DeepSeek R1 (original) / o3-mini · low effort
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
R1 0528
41.7%
1
USAMO 2025 · DeepSeek R1 (original)
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
R1 0528
4.8%
1
ArXivMath · January 2026 · GPT-5.2 · low effort
Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.2
47.9%
1
FrontierMath Tier
4
Models solve expert-written advanced mathematics problems on the named tier and edition
Epoch
Claude Opus 4.8
31.3%
33
FrontierMath Erdos
Models solve expert-written advanced mathematics problems on the named tier and edition
Epoch
GPT-6.1 Sol
4.4%
13
Science
5
Humanity's Last Exam
Very hard expert questions across many fields, written to stump today’s models
AA
Claude Opus 5.5
61.4%
337
CritPt (physics)
Research-level physics problems that need many careful reasoning steps
AA
Muse Wonder Full
32.9%
324
SciCode
Writing code for real scientific computing problems (physics, biology, math)
AA
Claude Opus 5.5
66.9%
140
Terminal-Bench Science
0.1
Agents complete expert-authored research workflows across five scientific domains in a terminal
AA
GPT-6 Astra
63.3%
33
Vals Science Index
Scientific problem solving across the index’s component evaluations
Vals
GPT-6 Astra
65.1%
18
Coding
14
Terminal-Bench
4.0
Agents use a terminal to complete executable coding, system administration and research tasks in the 4.0 suite
Claude Sonnet 5.5
63.6%
132
+1 versions & slices
2.1
Older, easier version of Terminal-Bench; Still widely reported
AA
Claude Fable 5.1
91.4%
156
Terminal-Bench Hard
Agents solve difficult software, system-administration and data-processing tasks in a terminal
AA
GPT-5.6 Sol
65.9%
242
WebDev Arena
Human pairwise preferences in the stated webdev vote pool
Arena
Claude Opus 5.5
1,807
97
+1 versions & slices
full cohort
Human pairwise preferences in the stated webdev vote pool
Arena
Claude Opus 5.5
1,807
97
DeepSWE
1.1
Agents change repository code to resolve software engineering tasks under the stated DeepSWE edition
GPT-6 Astra
74.1%
44
+1 versions & slices
1
Agents change repository code to resolve software engineering tasks under the stated DeepSWE edition
Datacurve
GPT-5.5
70%
20
ITBench SRE
IT incident tasks: diagnose and fix broken systems like a site-reliability engineer
AA
GPT-5.6 Sol
56.2%
37
FrontierCode
1.1
Agents implement changes in real repositories, graded against maintainer-defined requirements
AA
Claude Opus 5.5
57.1%
28
SWE-bench Pro (v2 full)
Agents resolve repository issues under the network-locked v2 protocol; full and 51-task Hard results are separate
Scale
Claude Opus 5
99.4%
16
SWE-bench Pro (v2 hard)
Agents resolve repository issues under the network-locked v2 protocol; full and 51-task Hard results are separate
Scale
Claude Opus 5
98%
16
SWE-bench Pro (private)
Agents resolve long-horizon professional repository tasks; public and private splits are separate
Scale
Muse Spark 1.1
51.5%
9
MirrorCode
Agents reimplement software projects by observing their behavior
Epoch
Claude Opus 5.5
77.4%
9
MultiLangBench
Models build a SQLite-compatible database engine in 20 programming languages
OpenAI
GPT-6 Astra
87.4%
8
Agents
16
GDPval-AA (work tasks)
2.1
Real work tasks from many professions (reports, spreadsheets, analyses), graded against expert work
AA
Claude Opus 5.5
68.3
175
AA-Briefcase
1.1 · Elo
Agents produce professional deliverables; blind pairwise comparisons produce the source Elo rating
AA
Claude Sonnet 5.5
1,817
140
τ³-Banking
AA
Agents retrieve banking knowledge and execute tools to resolve customer-support workflows
AA
Grok 4.6
50.7%
134
AutomationBench-AA
Agents orchestrate SaaS workflows through APIs; objectives receive credit only when the task has no guardrail violation
AA
Gemini 4 Argon
77.5%
133
EnterpriseOps-Gym-AA
Agents complete stateful enterprise workflows in eight domains; final database state determines success
AA
Claude Fable 5
51.1%
43
Furniture Assembly
Models inspect furniture assembly photos, identify incorrect builds and locate the mistaken step
Epoch
Claude Opus 5.5
83.3%
31
APEX-Agents
Agents complete long-horizon professional-services tasks across realistic applications
AA
Qwen3.8 Max (0902)
42.4%
29
EBR Bench
Agents learn to play Earthborne Rangers across repeated playthroughs
Epoch
GPT-6 Astra
76.2%
24
τ³-bench
banking · 1.0.1
τ-bench official · 3-banking-1.0.1: Customer-service agents converse with simulated users, retrieve domain knowledge and change state through tools
Sierra
Claude Opus 5
48.7%
21
MCP-Atlas Public
MCPAtlas Public: MCPAtlas Public as reported in this launch evaluation; the post is the authority for the recorded task and grading scope
Muse Spark
82.4%
16
MCP-Atlas
full 1000
Tool orchestration across MCP servers on the stated task split
Scale
Muse Spark
82.2%
16
SpireClimb
Agents play Slay the Spire under the named game and difficulty setting
Epoch
GPT-6 Astra
25.7%
7
SpireClimb StS1 A20 Heart
Agents play Slay the Spire under the named game and difficulty setting
Epoch
GPT-6.1 Sol
20%
7
SpireClimb StS2 A10
Agents play Slay the Spire under the named game and difficulty setting
Epoch
GPT-6 Astra
34.7%
7
Toolathlon-Verified
Toolathlon-Verified as reported in this launch evaluation; the post is the authority for the recorded task and grading scope
GLM 5.3 Flash
78.4%
6
AutomationBench
Zapier run
End-to-end completion of business workflows using real SaaS applications
Zapier
Gemini 4 Argon
51.3%
2
Multimodal
6
MMMU-Pro (vision)
College-level questions that require understanding images: charts, diagrams, photos
AA
Claude Opus 5.5
87.7%
127
Arena Vision
Human pairwise preferences in the stated vision vote pool
Arena
Claude Fable 5
1,325
88
MMMU
validation
Image-grounded university-level questions across 30 disciplines; validation and test sets remain separate
MMMU
GPT-5.1
85.4%
41
Chartography
Expert chart reasoning over professional scientific, medical, financial and engineering charts
Surge
Gemini 4 Argon
71.6%
37
OCRBench
v2 · CN
Tests visual text recognition, localization and reasoning; English and Chinese subsets remain separate
OCRBench
O
InternVL3 8B
53.5%
4
+1 versions & slices
v2 · EN
Tests visual text recognition, localization and reasoning; English and Chinese subsets remain separate
OCRBench
O
InternVL3 8B
49%
4
Writing
3
EQ-Bench Longform Creative Writing
1.11
Longform Creative Writing 1.11: Models plan, revise and write a story across eight turns, graded with a writing rubric
EQ-Bench
Claude Opus 5
86.3%
95
EQ-Bench Creative Writing
v3
Creative Writing v3 · Elo: Judge-rated creative writing under the v3 prompt set and pairwise ranking
EQ-Bench
GPT-6 Astra
2,173
91
Lech Mazur Short-Story
Judges compare stories written to ten required elements; the comparison score can be negative
Lech
Claude Fable 5.1
3.8
44
Health & law
7
Vals LegalBench
Legal reasoning and legal-language understanding on LegalBench
Vals
Claude Fable 5
88.6%
89
+1 versions & slices
DeepSeek V3 March 2025
Legal reasoning and legal-language understanding on LegalBench
Vals
DeepSeek V3
77.7%
1
Vals MedScribe
Accuracy of clinical notes produced from consultation transcripts
Vals
Claude Opus 5.5
91.4%
80
Vals MedCode
Medical coding accuracy on patient encounters
Vals
Claude Opus 5
63.6%
78
Vals Legal Research Bench
Grounded legal research; a task passes only if all criteria pass
Vals
Claude Fable 5.1
55.3%
69
MLCR-AA
Models synthesize long medical records; the headline credits concise answers judged both accurate and complete
AA
Claude Sonnet 5.5
75%
64
Harvey LAB-AA
1.1
Legal agents work from case documents; every rubric criterion must pass and material hallucinations receive no credit
AA
Muse Wonder Full
13.1%
23
Code review
1
Code Review Bench
Core F2
Review systems identify pull-request defects; core F2 weights recall more heavily than precision over the published gold set
Martian
Documents
4
AA-LCR (long context)
1.1
Questions that need reading and connecting facts across very long documents (around 100K tokens)
AA
Kimi K3
88.7%
275
GDP.pdf
AA all-pass
Models reason over professional PDF documents; an attempt passes only when every atomic criterion passes
Muse Wonder Full
32.6%
133
Arena Document
Human pairwise preferences in the stated document vote pool
Arena
Claude Opus 5
1,516
33
OmniDocBench
v1.6 · full
End-to-end document parsing, averaging text recognition, table structure and formula recognition under the stated full-set protocol
OmniDocBench
X
TeleOCR
96.9%
22
Other
12
Community Decision Index
0.4.1
Decision models answer typed questions across knowledge, language, retrieval, tools and arts tasks
Community
b
mammoth-1 (27B)
68.9
99
AA Decision
Humanity's Last Exam text multiple-choice accuracy
Decision models select answers to text-only Humanity’s Last Exam multiple-choice questions
AA
T
Jev 1.13
20.9%
15
+4 versions & slices
Omniscience accuracy
Decision models select correct answers to factual knowledge questions
AA
D1
41%
15
Omniscience non-hallucination rate
Decision models answer factual questions, scored on avoiding incorrect assertions
AA
D1 Ultra
84.5%
15
When2Call accuracy
Decision models decide when a request requires a tool call
AA
D1 Ultra
92.6%
15
WildGuardTest average F1
Decision models classify safety-related inputs, scored by average F1
AA
D1 Ultra
95.4%
15
AA Decision cost per task
v1.0
Decision models run the AA decision workload, measured by cost per task
AA
Decider Flash v1.2
0
15
AA Decision Index
v1.0
Decision models answer factual and multiple-choice questions and classify tool-use and safety cases in the AA composite
AA
D1 Ultra
65.2
15
AA Decision time per task
v1.0
Decision models run the AA decision workload, measured by elapsed time per task
AA
C
Celeris-1 Decision
63
15
JevBench
v1.6.1
Decision models produce typed choices, nulls and scores, evaluated by the stated intelligence, calibration, speed and cost composite
Benchmark
Mercury Decide
72.4
5
+2 versions & slices
v1.6.37 · open capability
Decision models produce typed choices, nulls and scores, evaluated on intelligence and calibration within the stated cost and latency caps
Benchmark
J
Kev 27B
78.9
5
v1.6.37 · api composite
Decision models produce typed choices, nulls and scores, evaluated by the stated intelligence, calibration, speed and cost composite
Benchmark
Mercury Decide
72.4
3
Legacy · saturated or no longer reported
38
GPQA Diamond
Graduate-level biology, chemistry and physics questions that can’t be solved by a quick web search
GPT-6 Astra
96.3%
315
+2 versions & slices
Epoch
Models answer graduate-level science multiple-choice questions
Epoch
Gemini 4 Argon
96.4%
133
January 2025 run
Models answer graduate-level science multiple-choice questions
Epoch
DeepSeek V3
56.5%
1
τ²-bench
AA
Tests service agents that use tools while coordinating with a simulated user
AA
GLM 5.2
99.1%
245
+8 versions & slices
airline · 0.1.3
τ-bench official · 2-airline-0.1.3: Customer-service agents converse with simulated users, retrieve domain knowledge and change state through tools
Claude Sonnet 4.5
70%
8
retail · 0.1.3
τ-bench official · 2-retail-0.1.3: Customer-service agents converse with simulated users, retrieve domain knowledge and change state through tools
Claude Sonnet 4.5
86.2%
8
overall · 0.1.3
τ-bench official · 2-overall-0.1.3: Customer-service agents converse with simulated users, retrieve domain knowledge and change state through tools
Claude Sonnet 4.5
84.7%
7
telecom · 0.1.3
τ-bench official · 2-telecom-0.1.3: Customer-service agents converse with simulated users, retrieve domain knowledge and change state through tools
Qwen3 Max Thinking
98.2%
7
airline · 1.0.1
τ-bench official · 2-airline-1.0.1: Customer-service agents converse with simulated users, retrieve domain knowledge and change state through tools
Sierra
Claude Opus 4.5
84%
6
overall · 1.0.1
τ-bench official · 2-overall-1.0.1: Customer-service agents converse with simulated users, retrieve domain knowledge and change state through tools
Sierra
Qwen3.5 397B A17B
87.9%
6
retail · 1.0.1
τ-bench official · 2-retail-1.0.1: Customer-service agents converse with simulated users, retrieve domain knowledge and change state through tools
Sierra
Qwen3.5 397B A17B
84.4%
6
telecom · 1.0.1
τ-bench official · 2-telecom-1.0.1: Customer-service agents converse with simulated users, retrieve domain knowledge and change state through tools
Sierra
Qwen3.5 397B A17B
97.8%
6
LiveCodeBench
AA archived
AA evaluates solutions to newly collected competitive-programming problems
AA
Gemini 3 Flash Preview
90.8%
153
+1 versions & slices
official artifact (through April 2025)
Mean pass@1 over each model’s available problems in the current official public generation artifact; its latest problem is dated April 7, 2025
LiveCodeBench
o3
84.7%
10
AIME
2025 · AA archived
AA evaluates the 30 integer-answer problems from the 2025 AIME mathematics competition
AA
GPT-5.2
99%
140
OTIS Mock AIME 2024
2025
Models solve the stated OTIS mock AIME mathematics problems
Epoch
Claude Fable 5
100%
130
+1 versions & slices
2025 · February 2025 run
Models solve the stated OTIS mock AIME mathematics problems
Epoch
DeepSeek V3
15.8%
1
BFCL
v4
Tests function calling, multi-turn interactions, search and memory in the v4 suite
Berkeley
Claude Opus 4.5
77.5%
39
+2 versions & slices
v4 · prompt mode
Tests function calling, multi-turn interactions, search and memory in the v4 suite
Berkeley
Gemini 2.5 Flash
50.9%
16
v4 · native function calls
Tests function calling, multi-turn interactions, search and memory in the v4 suite
Berkeley
o3 (2025-04-16)
48.6%
2
MMLU-Pro
official board
Graduate-level multiple-choice knowledge and reasoning across 14 subjects, including science, mathematics, law and health
Gemini 3.1 Pro
91.2%
32
SWE-bench Verified
Epoch
Swe Bench Verified: Agents modify real repositories to resolve the stated Verified issues
Epoch
Claude Opus 4.7
83.5%
24
SWE-bench Pro (public)
Agents resolve long-horizon professional repository tasks; public and private splits are separate
Muse Spark 1.1
61.5%
19
Video-MME
2 · without subtitles/audio
Grouped non-linear question scoring for video understanding; this is distinct from mean question accuracy
Video-MME
Gemini 3 Flash Preview
32.9%
18
+6 versions & slices
2 · with subtitles/audio
Grouped non-linear question scoring for video understanding; this is distinct from mean question accuracy
Video-MME
Gemini 3 Flash Preview
42.5%
18
2 · without subtitles/audio · 512 frames
Grouped non-linear question scoring for video understanding; this is distinct from mean question accuracy
Video-MME
Kimi K2.5
33%
2
1 · without subtitles/audio
Question accuracy across short, medium and long videos
Video-MME
r
Aria
67.6%
1
1 · with subtitles/audio
Question accuracy across short, medium and long videos
Video-MME
r
Aria
72.1%
1
2 · with subtitles/audio · 512 frames
Grouped non-linear question scoring for video understanding; this is distinct from mean question accuracy
Video-MME
Kimi K2.5
40.7%
1
2 · with subtitles/audio · native audio · 512 frames
Grouped non-linear question scoring for video understanding; this is distinct from mean question accuracy
Video-MME
Qwen3.5 Omni Plus
27.4%
1
Aider Polyglot
two attempts
Models edit code for 225 Exercism exercises in six languages, with a second attempt after test feedback
Aider
GPT-5
88%
15
+1 versions & slices
two attempts · diff edits · December 2024 run
Models edit code for 225 Exercism exercises in six languages, with a second attempt after test feedback
Aider
Qwen2.5 Coder 32B Instruct
8%
1
SWE-rebench
all · 2026-02-01 to 2026-07-01
Repository issue resolution over the exact fresh pull-request window, under the source agent version
Nebius
Qwen 3.5 35B A3B
29.5%
1
+5 versions & slices
Go · 2026-02-01 to 2026-07-01
Repository issue resolution over the exact fresh pull-request window, under the source agent version
Nebius
Qwen 3.5 35B A3B
18.1%
1
Java · 2026-02-01 to 2026-07-01
Repository issue resolution over the exact fresh pull-request window, under the source agent version
Nebius
Qwen 3.5 35B A3B
16.2%
1
Python · 2026-02-01 to 2026-07-01
Repository issue resolution over the exact fresh pull-request window, under the source agent version
Nebius
Qwen 3.5 35B A3B
42.6%
1
Rust · 2026-02-01 to 2026-07-01
Repository issue resolution over the exact fresh pull-request window, under the source agent version
Nebius
Qwen 3.5 35B A3B
22.4%
1
TypeScript · 2026-02-01 to 2026-07-01
Repository issue resolution over the exact fresh pull-request window, under the source agent version
Nebius
Qwen 3.5 35B A3B
16.7%
1
No benchmark matches
Model
AA Intelligence Index 4.3.2
Per task
Price
Speed
Supports
Context
T
Inkling Small
Thinking Machines
Open
25.7
thinking
$0.089
$0.45
/
$1.20
219
256K
T
Inkling
Thinking Machines
Open
25
xhigh
—
$0.95
/
$4.05
190
256K