Model Menu
Models
Compare
Leaderboard
Benchmarks
⌘K
Top
Chat
Coding
Agents
Writing
Vision
Image
Video
Speech
Embeddings
Decision
Music
Filters
1
All AI models
All models
2292
Chat & Knowledge
558
Coding
381
Agents & Tool Use
541
Writing & Language
360
Vision & Documents
395
Image Generation & Editing
183
Video Generation
148
Speech & Voice
358
Embeddings & Retrieval
671
Decision models
118
Music & Sound
29
AA Intelligence Index 4.3.2
Humanity's Last Exam
CritPt
AA Omniscience · Accuracy
AA Omniscience Index
GPQA Diamond
All benchmarks
196
All benchmarks
196
×
All
196
Overall
1
Knowledge
27
Reasoning
9
Math
56
Science
9
Coding
26
Agents
28
Multimodal
13
Writing
3
Health & law
7
Code review
1
Documents
4
Other
12
Benchmark
By
Leader
Models
Overall
1
AA Intelligence Index
4.3.2
AA Intelligence Index 4.3.2: Artificial Analysis combines tests of knowledge, reasoning, coding and tool use into one score
AA
Claude Opus 5.5
57.6
337
Knowledge
27
AA Omniscience
Accuracy
AA Omniscience · Accuracy: Measures correct answers on the AA Omniscience factual knowledge test
AA
Claude Fable 5.1
67.2%
321
+1 versions & slices
Hallucination Rate
AA Omniscience · Hallucination Rate: Measures how often attempted answers are incorrect on the AA Omniscience factual knowledge test
AA
O
MiniCPM5 1B
0.9%
321
AA Omniscience Index
AA Omniscience Index: Scores factual knowledge while penalizing incorrect answers to encourage admitting uncertainty
AA
Claude Opus 5.5
46.4
321
IFBench (instructions)
IFBench (instructions): Checks whether the model follows precise formatting and content instructions
AA
Grok 4.3
83.3%
248
Arena Text
Arena Text · overall: Human pairwise preferences in the stated text vote pool
Arena
Gemini 4 Argon
1,524
196
+16 versions & slices
creative writing
Arena Text · creative writing: Human pairwise preferences in the stated text vote pool
Arena
Claude Opus 5.5
1,519
195
English
Arena Text · english: Human pairwise preferences in the stated text vote pool
Arena
Gemini 4 Argon
1,538
188
instruction following
Arena Text · instruction following: Human pairwise preferences in the stated text vote pool
Arena
Claude Opus 5.5
1,536
188
non-English
Arena Text · non english: Human pairwise preferences in the stated text vote pool
Arena
Gemini 4 Argon
1,522
188
overall · full cohort
Arena Text · overall · snapshot 2026-10-15 · native board settings: Human pairwise preferences in the stated text vote pool
Arena
Gemini 4 Argon
1,532
188
creative writing · full cohort
Arena Text · creative writing · snapshot 2026-10-15 · native board settings: Human pairwise preferences in the stated text vote pool
Arena
Claude Opus 5.5
1,534
187
Russian
Arena Text · russian: Human pairwise preferences in the stated text vote pool
Arena
Gemini 4 Argon
1,534
186
Chinese
Arena Text · chinese: Human pairwise preferences in the stated text vote pool
Arena
Gemini 4 Argon
1,608
181
Spanish
Arena Text · spanish: Human pairwise preferences in the stated text vote pool
Arena
Claude Fable 5.1
1,521
155
German
Arena Text · german: Human pairwise preferences in the stated text vote pool
Arena
Claude Opus 5
1,526
154
French
Arena Text · french: Human pairwise preferences in the stated text vote pool
Arena
Claude Fable 5.1
1,527
149
Korean
Arena Text · korean: Human pairwise preferences in the stated text vote pool
Arena
Claude Fable 5.1
1,521
144
Japanese
Arena Text · japanese: Human pairwise preferences in the stated text vote pool
Arena
Claude Fable 5.1
1,560
138
Polish
Arena Text · polish: Human pairwise preferences in the stated text vote pool
Arena
Claude Fable 5.1
1,524
133
creative writing · Gemini 1.5 Flash (001) / Gemini 1.5 Pro (001) / Mistral Large 2402
Arena Text · creative writing · Arena creative; source adjustment/filter settings · historical run: Human pairwise preferences in the stated text vote pool
Arena
Mistral Large
1,211
1
overall · Gemini 1.5 Flash (001) / Gemini 1.5 Pro (001) / Mistral Large 2402
Arena Text · overall · Arena text; source adjustment/filter settings · historical run: Human pairwise preferences in the stated text vote pool
Arena
Mistral Large
1,243
1
SimpleQA Verified
v1
SimpleQA Verified v1: Accuracy on short factual questions with verified answers, answered without web access
Gemini 3.1 Pro Preview
77.5%
83
+1 versions & slices
v1 · October 2026 run
SimpleQA Verified v1 · Kaggle SimpleQA Verified v1; generation settings not stated · historical run: Accuracy on short factual questions with verified answers, answered without web access
Gemma 4 31B
10.7%
1
Mystery Game Puzzles
Mystery Game Puzzles: Models choose the best move in puzzle positions from an undisclosed game
Epoch
GPT-6 Astra
84%
64
+1 versions & slices
August 2026 run
Mystery Game Puzzles · Epoch independent run · historical run: Models choose the best move in puzzle positions from an undisclosed game
Epoch
Gemini 3.1 Pro Preview
29%
1
EQ-Bench
3
EQ-Bench 3: Judge-rated responses to social and emotionally nuanced scenarios, scored through the source pairwise ranking
EQ-Bench
Claude Fable 5
2,050
50
+1 versions & slices
4
Active emotional and social intelligence in multi-turn conversations with 120 simulated personas
EQ-Bench
Kimi K3
1,339
1
Reasoning
9
Chess Puzzles
Chess Puzzles: Models choose the best next move from a novel chess position
Epoch
GPT-6 Astra
74%
68
ARC-AGI-2
semi-private
ARC-AGI-2 · semi-private: Models infer transformations from examples and solve unseen grid puzzles; the evaluation split is explicit
ARC
GPT-6 Astra
95%
66
+3 versions & slices
public evaluation
ARC-AGI-2 · public-eval: Models infer transformations from examples and solve unseen grid puzzles; the evaluation split is explicit
ARC
Claude Fable 5.1
99.2%
63
public evaluation · chain of thought
ARC-AGI-2 · public-eval · ARC CoT · historical run: Models infer transformations from examples and solve unseen grid puzzles; the evaluation split is explicit
ARC
R1 0528
0.3%
1
semi-private · chain of thought
ARC-AGI-2 · semi-private · ARC CoT · historical run: Models infer transformations from examples and solve unseen grid puzzles; the evaluation split is explicit
ARC
R1 0528
1.1%
1
ARC-AGI-1
semi-private
ARC-AGI-1 · semi-private: Models infer transformations from examples and solve unseen grid puzzles; the evaluation split is explicit
ARC
Claude Fable 5
98.5%
64
+2 versions & slices
public evaluation
ARC-AGI-1 · public-eval: Models infer transformations from examples and solve unseen grid puzzles; the evaluation split is explicit
ARC
Gemini 4 Argon
99.5%
61
semi-private · chain of thought
ARC-AGI-1 · semi-private · ARC CoT · historical run: Models infer transformations from examples and solve unseen grid puzzles; the evaluation split is explicit
ARC
R1 0528
21.2%
1
ARC-AGI-3
semi-private
ARC-AGI-3 · semi-private: Measures interactive game solving through relative human action efficiency
ARC
GPT-6 Astra
62.7%
18
Math
53
FrontierMath Tiers 1–3 (v2)
FrontierMath Tiers 1–3 (v2): Models solve expert-written advanced mathematics problems on the named tier and edition
Epoch
GPT-6 Astra
93.7%
69
FrontierMath Tier 4 (v2)
FrontierMath Tier 4 (v2): Models solve expert-written advanced mathematics problems on the named tier and edition
Epoch
GPT-6.1 Sol
100%
54
AA Analyst Agent
AA Analyst Agent: Data-analyst tasks: the model explores data with tools and reports findings
AA
Claude Fable 5.1
57.5%
43
FrontierMath
FrontierMath: Models solve expert-written advanced mathematics problems on the named tier and edition
Epoch
GPT-5.4 Pro (2026-03-05)
50%
40
MATH Level
5
MATH Level 5: Models solve competition mathematics problems from the hardest MATH difficulty level
Epoch
GPT-5 (2025-08-07)
98.1%
35
MathArena
Apex
MathArena · apex · Apex: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
Claude Opus 4.8
81.3%
34
+45 versions & slices
AIME 2025
MathArena · aime · AIME 2025: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.2
100%
30
Apex Shortlist
MathArena · apex · Apex Shortlist: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.5
98.4%
30
HMMT February 2025
MathArena · hmmt · HMMT Feb 2025: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.2
100%
30
AIME 2026
MathArena · AIME 2026: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
Claude Opus 4.8
100%
24
HMMT February 2026
MathArena · HMMT February 2026: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.5
98.5%
24
ArXivMath · June 2026
MathArena · arxiv · 06/2026: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-6 Astra
94.4%
24
BrokenArXiv · June 2026
MathArena · arxiv_false · 06/2026: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
Claude Opus 5.5
100%
24
BRUMO 2025
MathArena · brumo · BRUMO 2025: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
Gemini 3 Flash Preview
100%
24
SMT 2025
MathArena · smt · SMT 2025: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.2
96%
24
CMIMC 2025
MathArena · cmimc · CMIMC 2025: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.2
95.6%
22
ArXivMath · February 2026
MathArena · arxiv · 02/2026: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.4 Pro
75.8%
20
ArXivMath · May 2026
MathArena · arxiv · 05/2026: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-6 Astra
95%
19
ArXivMath · January 2026
MathArena · arxiv · 01/2026: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.4
76.1%
18
BrokenArXiv · May 2026
MathArena · arxiv_false · 05/2026: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
Claude Opus 5.5
95%
18
ArXivMath · April 2026
MathArena · arxiv · 04/2026: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-6 Astra
90.8%
17
Kangaroo 2025 1-2
MathArena · kangaroo · Kangaroo 2025 1-2: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.5
95.8%
17
Kangaroo 2025 11-12
MathArena · kangaroo · Kangaroo 2025 11-12: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
Claude Opus 4.8
100%
17
Kangaroo 2025 3-4
MathArena · kangaroo · Kangaroo 2025 3-4: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.5
89.6%
17
Kangaroo 2025 5-6
MathArena · kangaroo · Kangaroo 2025 5-6: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.5
90%
17
Kangaroo 2025 7-8
MathArena · kangaroo · Kangaroo 2025 7-8: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
Claude Opus 4.8
96.7%
16
Kangaroo 2025 9-10
MathArena · kangaroo · Kangaroo 2025 9-10: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
Gemini 3.1 Pro Preview
100%
16
BrokenArXiv · April 2026
MathArena · arxiv_false · 04/2026: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-6 Astra
97.5%
15
BrokenArXiv · February 2026
MathArena · arxiv_false · 02/2026: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.5
68.2%
15
ArXivMath · March 2026
MathArena · arxiv · 03/2026: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.5
77.5%
14
HMMT November 2025
MathArena · hmmt · HMMT Nov 2025: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.2
99.2%
14
ArXivMath · December 2025
MathArena · arxiv · 12/2025: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
Gemini 3.1 Pro Preview
66.2%
13
BrokenArXiv · March 2026
MathArena · arxiv_false · 03/2026: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.5
73.7%
13
ArXivMath · August 2026
MathArena · arxiv · 08/2026: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-6.1 Sol
96.5%
12
BrokenArXiv · August 2026
MathArena · arxiv_false · 08/2026: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-6.1 Sol
86.3%
12
Project Euler
MathArena · euler · Project Euler: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
Gemini 3.1 Pro Preview
89%
11
ArXivLean · June 2026
MathArena · arxivlean · 06/2026: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-6 Astra
65.2%
9
USAMO 2026
MathArena · usamo · USAMO 2026: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.5
98.2%
8
ArXivMath · September 2026
MathArena · arxiv · 09/2026: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-6.1 Sol
91.5%
7
BrokenArXiv · September 2026
MathArena · arxiv_false · 09/2026: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-6.1 Sol
82.7%
7
ArXivLean · March 2026
MathArena · arxivlean · 03/2026: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.4
17.1%
5
ArXivLean · September 2026
MathArena · arxivlean · 09/2026: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
Claude Opus 5.5
47.2%
5
IMO 2025
MathArena · imo · IMO 2025: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5
38.1%
4
Putnam 2025
MathArena · putnam · Putnam 2025: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5 Pro
77.5%
3
USAMO 2025
MathArena · usamo · USAMO 2025: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
R1 0528
30.1%
3
Miklós Schweitzer 2025
MathArena · miklos · Miklós Schweitzer 2025: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5 Pro
90%
1
AIME 2025 · DeepSeek R1 (original) / o3-mini · low effort
MathArena · aime · AIME 2025 · MathArena competition run · historical run: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
R1 0528
70%
1
BRUMO 2025 · DeepSeek R1 (original)
MathArena · brumo · BRUMO 2025 · MathArena competition run · historical run: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
R1 0528
80.8%
1
HMMT February 2025 · DeepSeek R1 (original) / o3-mini · low effort
MathArena · hmmt · HMMT Feb 2025 · MathArena competition run · historical run: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
R1 0528
41.7%
1
USAMO 2025 · DeepSeek R1 (original)
MathArena · usamo · USAMO 2025 · MathArena competition run · historical run: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
R1 0528
4.8%
1
ArXivMath · January 2026 · GPT-5.2 · low effort
MathArena · arxiv · 01/2026 · MathArena competition run · historical run: Answer accuracy on the named mathematics competition under the recorded effort
MathArena
GPT-5.2
47.9%
1
FrontierMath Tier
4
FrontierMath Tier 4: Models solve expert-written advanced mathematics problems on the named tier and edition
Epoch
Claude Opus 4.8
31.3%
33
FrontierMath Erdos
FrontierMath Erdos: Models solve expert-written advanced mathematics problems on the named tier and edition
Epoch
GPT-6.1 Sol
4.4%
13
Science
5
Humanity's Last Exam
Humanity's Last Exam: Very hard expert questions across many fields, written to stump today’s models
AA
Claude Opus 5.5
61.4%
337
CritPt (physics)
CritPt (physics): Research-level physics problems that need many careful reasoning steps
AA
Muse Wonder Full
32.9%
324
SciCode
SciCode: Writing code for real scientific computing problems (physics, biology, math)
AA
Claude Opus 5.5
66.9%
140
Terminal-Bench Science
0.1
Terminal-Bench Science 0.1: Agents complete expert-authored research workflows across five scientific domains in a terminal
AA
GPT-6 Astra
63.3%
33
Vals Science Index
Vals Science Index: Scientific problem solving across the index’s component evaluations
Vals
GPT-6 Astra
65.1%
18
Coding
14
Terminal-Bench Hard
Terminal-Bench Hard (version unstated): Agents solve difficult software, system-administration and data-processing tasks in a terminal
AA
GPT-5.6 Sol
65.9%
242
Terminal-Bench
2.1
Terminal-Bench 2.1 · source setting 2: Older, easier version of Terminal-Bench; Still widely reported
AA
Claude Fable 5.1
91.4%
156
+1 versions & slices
4.0
Terminal-Bench 4.0 · source setting 2: Agents use a terminal to complete executable coding, system administration and research tasks in the 4.0 suite
Claude Sonnet 5.5
63.6%
132
WebDev Arena
WebDev Arena: Human pairwise preferences in the stated webdev vote pool
Arena
Claude Opus 5.5
1,807
97
+1 versions & slices
full cohort
WebDev Arena · snapshot 2026-10-19 · native board settings: Human pairwise preferences in the stated webdev vote pool
Arena
Claude Opus 5.5
1,807
97
DeepSWE
1.1
DeepSWE 1.1: Agents change repository code to resolve software engineering tasks under the stated DeepSWE edition
GPT-6 Astra
74.1%
44
+1 versions & slices
1
DeepSWE 1: Agents change repository code to resolve software engineering tasks under the stated DeepSWE edition
Datacurve
GPT-5.5
70%
20
ITBench SRE
ITBench SRE: IT incident tasks: diagnose and fix broken systems like a site-reliability engineer
AA
GPT-5.6 Sol
56.2%
37
FrontierCode
1.1
FrontierCode 1.1: Agents implement changes in real repositories, graded against maintainer-defined requirements
AA
Claude Opus 5.5
57.1%
28
SWE-bench Pro (v2 full)
SWE-bench Pro (v2 full): Agents resolve repository issues under the network-locked v2 protocol; full and 51-task Hard results are separate
Scale
Claude Opus 5
99.4%
16
SWE-bench Pro (v2 hard)
SWE-bench Pro (v2 hard): Agents resolve repository issues under the network-locked v2 protocol; full and 51-task Hard results are separate
Scale
Claude Opus 5
98%
16
SWE-bench Pro (private)
SWE-bench Pro (private): Agents resolve long-horizon professional repository tasks; public and private splits are separate
Scale
Muse Spark 1.1
51.5%
9
MirrorCode
MirrorCode: Agents reimplement software projects by observing their behavior
Epoch
Claude Opus 5.5
77.4%
9
MultiLangBench
MultiLangBench: Models build a SQLite-compatible database engine in 20 programming languages
OpenAI
GPT-6 Astra
87.4%
8
Agents
16
GDPval-AA (work tasks)
2.1
GDPval-AA (work tasks) 2.1: Real work tasks from many professions (reports, spreadsheets, analyses), graded against expert work
AA
Claude Opus 5.5
68.3
175
AA-Briefcase
1.1 · Elo
AA-Briefcase 1.1 · Elo: Agents produce professional deliverables; blind pairwise comparisons produce the source Elo rating
AA
Claude Sonnet 5.5
1,817
140
τ³-Banking
AA
τ³-Banking · AA: Agents retrieve banking knowledge and execute tools to resolve customer-support workflows
AA
Grok 4.6
50.7%
134
AutomationBench-AA
AutomationBench-AA: Agents orchestrate SaaS workflows through APIs; objectives receive credit only when the task has no guardrail violation
AA
Gemini 4 Argon
77.5%
133
EnterpriseOps-Gym-AA
EnterpriseOps-Gym-AA: Agents complete stateful enterprise workflows in eight domains; final database state determines success
AA
Claude Fable 5
51.1%
43
Furniture Assembly
Furniture Assembly: Models inspect furniture assembly photos, identify incorrect builds and locate the mistaken step
Epoch
Claude Opus 5.5
83.3%
31
APEX-Agents
APEX-Agents: Agents complete long-horizon professional-services tasks across realistic applications
AA
Qwen3.8 Max (0902)
42.4%
29
EBR Bench
EBR Bench: Agents learn to play Earthborne Rangers across repeated playthroughs
Epoch
GPT-6 Astra
76.2%
24
τ³-bench
banking · 1.0.1
τ-bench official · 3-banking-1.0.1: Customer-service agents converse with simulated users, retrieve domain knowledge and change state through tools
Sierra
Claude Opus 5
48.7%
21
MCP-Atlas Public
MCPAtlas Public: MCPAtlas Public as reported in this launch evaluation; the post is the authority for the recorded task and grading scope
Muse Spark
82.4%
16
MCP-Atlas
full 1000
MCP-Atlas · full 1000: Tool orchestration across MCP servers on the stated task split
Scale
Muse Spark
82.2%
16
SpireClimb
SpireClimb: Agents play Slay the Spire under the named game and difficulty setting
Epoch
GPT-6 Astra
25.7%
7
SpireClimb StS1 A20 Heart
SpireClimb StS1 A20 Heart: Agents play Slay the Spire under the named game and difficulty setting
Epoch
GPT-6.1 Sol
20%
7
SpireClimb StS2 A10
SpireClimb StS2 A10: Agents play Slay the Spire under the named game and difficulty setting
Epoch
GPT-6 Astra
34.7%
7
Toolathlon-Verified
Toolathlon-Verified: Toolathlon-Verified as reported in this launch evaluation; the post is the authority for the recorded task and grading scope
GLM 5.3 Flash
78.4%
6
AutomationBench
Zapier run
AutomationBench · Zapier run: End-to-end completion of business workflows using real SaaS applications
Zapier
Gemini 4 Argon
51.3%
2
Multimodal
6
MMMU-Pro (vision)
MMMU-Pro (vision): College-level questions that require understanding images: charts, diagrams, photos
AA
Claude Opus 5.5
87.7%
127
Arena Vision
Arena Vision · overall: Human pairwise preferences in the stated vision vote pool
Arena
Claude Fable 5
1,325
88
MMMU
validation
MMMU · validation: Image-grounded university-level questions across 30 disciplines; validation and test sets remain separate
MMMU
GPT-5.1
85.4%
41
Chartography
Chartography: Expert chart reasoning over professional scientific, medical, financial and engineering charts
Surge
Gemini 4 Argon
71.6%
37
OCRBench
v2 · CN
OCRBench v2 · CN: Tests visual text recognition, localization and reasoning; English and Chinese subsets remain separate
OCRBench
O
InternVL3 8B
53.5%
4
+1 versions & slices
v2 · EN
OCRBench v2 · EN: Tests visual text recognition, localization and reasoning; English and Chinese subsets remain separate
OCRBench
O
InternVL3 8B
49%
4
Writing
3
EQ-Bench Longform Creative Writing
1.11
Longform Creative Writing 1.11: Models plan, revise and write a story across eight turns, graded with a writing rubric
EQ-Bench
Claude Opus 5
86.3%
95
EQ-Bench Creative Writing
v3
Creative Writing v3 · Elo: Judge-rated creative writing under the v3 prompt set and pairwise ranking
EQ-Bench
GPT-6 Astra
2,173
91
Lech Mazur Short-Story
Lech Mazur Short-Story · comparison score: Judges compare stories written to ten required elements; the comparison score can be negative
Lech
Claude Fable 5.1
3.8
44
Health & law
7
Vals LegalBench
Vals LegalBench: Legal reasoning and legal-language understanding on LegalBench
Vals
Claude Fable 5
88.6%
89
+1 versions & slices
DeepSeek V3 March 2025
Vals LegalBench · Vals default evaluation · historical run: Legal reasoning and legal-language understanding on LegalBench
Vals
DeepSeek V3
77.7%
1
Vals MedScribe
Vals MedScribe: Accuracy of clinical notes produced from consultation transcripts
Vals
Claude Opus 5.5
91.4%
80
Vals MedCode
Vals MedCode: Medical coding accuracy on patient encounters
Vals
Claude Opus 5
63.6%
78
Vals Legal Research Bench
Vals Legal Research Bench: Grounded legal research; a task passes only if all criteria pass
Vals
Claude Fable 5.1
55.3%
69
MLCR-AA
MLCR-AA: Models synthesize long medical records; the headline credits concise answers judged both accurate and complete
AA
Claude Sonnet 5.5
75%
64
Harvey LAB-AA
1.1
Harvey LAB-AA 1.1: Legal agents work from case documents; every rubric criterion must pass and material hallucinations receive no credit
AA
Muse Wonder Full
13.1%
23
Code review
1
Code Review Bench
Core F2
Review systems identify pull-request defects; core F2 weights recall more heavily than precision over the published gold set
Martian
Documents
4
AA-LCR (long context)
1.1
AA-LCR (long context) 1.1: Questions that need reading and connecting facts across very long documents (around 100K tokens)
AA
Kimi K3
88.7%
275
GDP.pdf
AA all-pass
GDP.pdf · AA all-pass: Models reason over professional PDF documents; an attempt passes only when every atomic criterion passes
Muse Wonder Full
32.6%
133
Arena Document
Arena Document · overall: Human pairwise preferences in the stated document vote pool
Arena
Claude Opus 5
1,516
33
OmniDocBench
v1.6 · full
OmniDocBench · v1.6 full: End-to-end document parsing, averaging text recognition, table structure and formula recognition under the stated full-set protocol
OmniDocBench
X
TeleOCR
96.9%
22
Other
12
Community Decision Index
0.4.1
Community Decision Index 0.4.1: Decision models answer typed questions across knowledge, language, retrieval, tools and arts tasks
Community
b
mammoth-1 (27B)
68.9
99
AA Decision
Humanity's Last Exam text multiple-choice accuracy
AA Decision · HLE text MCQ accuracy: Decision models select answers to text-only Humanity’s Last Exam multiple-choice questions
AA
T
Jev 1.13
20.9%
15
+4 versions & slices
Omniscience accuracy
AA Decision · Omniscience accuracy: Decision models select correct answers to factual knowledge questions
AA
D1
41%
15
Omniscience non-hallucination rate
AA Decision · Omniscience non-hallucination rate: Decision models answer factual questions, scored on avoiding incorrect assertions
AA
D1 Ultra
84.5%
15
When2Call accuracy
AA Decision · When2Call accuracy: Decision models decide when a request requires a tool call
AA
D1 Ultra
92.6%
15
WildGuardTest average F1
AA Decision · WildGuardTest average F1: Decision models classify safety-related inputs, scored by average F1
AA
D1 Ultra
95.4%
15
AA Decision cost per task
v1.0
AA Decision cost per task v1.0: Decision models run the AA decision workload, measured by cost per task
AA
Decider Flash v1.2
0
15
AA Decision Index
v1.0
AA Decision Index v1.0: Decision models answer factual and multiple-choice questions and classify tool-use and safety cases in the AA composite
AA
D1 Ultra
65.2
15
AA Decision time per task
v1.0
AA Decision time per task v1.0: Decision models run the AA decision workload, measured by elapsed time per task
AA
C
Celeris-1 Decision
63
15
JevBench
v1.6.1
JevBench v1.6.1: Decision models produce typed choices, nulls and scores, evaluated by the stated intelligence, calibration, speed and cost composite
Benchmark
Mercury Decide
72.4
5
+2 versions & slices
v1.6.37 · open capability
JevBench v1.6.37 · open capability: Decision models produce typed choices, nulls and scores, evaluated on intelligence and calibration within the stated cost and latency caps
Benchmark
J
Kev 27B
78.9
5
v1.6.37 · api composite
JevBench v1.6.37 · api composite: Decision models produce typed choices, nulls and scores, evaluated by the stated intelligence, calibration, speed and cost composite
Benchmark
Mercury Decide
72.4
3
Legacy · saturated or no longer reported
38
GPQA Diamond
GPQA Diamond: Graduate-level biology, chemistry and physics questions that can’t be solved by a quick web search
GPT-6 Astra
96.3%
315
+2 versions & slices
Epoch
GPQA Diamond · source setting 2: Models answer graduate-level science multiple-choice questions
Epoch
Gemini 4 Argon
96.4%
133
January 2025 run
GPQA Diamond · Epoch independent run · historical run: Models answer graduate-level science multiple-choice questions
Epoch
DeepSeek V3
56.5%
1
τ²-bench
AA
τ²-Bench: Tests service agents that use tools while coordinating with a simulated user
AA
GLM 5.2
99.1%
245
+8 versions & slices
airline · 0.1.3
τ-bench official · 2-airline-0.1.3: Customer-service agents converse with simulated users, retrieve domain knowledge and change state through tools
Claude Sonnet 4.5
70%
8
retail · 0.1.3
τ-bench official · 2-retail-0.1.3: Customer-service agents converse with simulated users, retrieve domain knowledge and change state through tools
Claude Sonnet 4.5
86.2%
8
overall · 0.1.3
τ-bench official · 2-overall-0.1.3: Customer-service agents converse with simulated users, retrieve domain knowledge and change state through tools
Claude Sonnet 4.5
84.7%
7
telecom · 0.1.3
τ-bench official · 2-telecom-0.1.3: Customer-service agents converse with simulated users, retrieve domain knowledge and change state through tools
Qwen3 Max Thinking
98.2%
7
airline · 1.0.1
τ-bench official · 2-airline-1.0.1: Customer-service agents converse with simulated users, retrieve domain knowledge and change state through tools
Sierra
Claude Opus 4.5
84%
6
overall · 1.0.1
τ-bench official · 2-overall-1.0.1: Customer-service agents converse with simulated users, retrieve domain knowledge and change state through tools
Sierra
Qwen3.5 397B A17B
87.9%
6
retail · 1.0.1
τ-bench official · 2-retail-1.0.1: Customer-service agents converse with simulated users, retrieve domain knowledge and change state through tools
Sierra
Qwen3.5 397B A17B
84.4%
6
telecom · 1.0.1
τ-bench official · 2-telecom-1.0.1: Customer-service agents converse with simulated users, retrieve domain knowledge and change state through tools
Sierra
Qwen3.5 397B A17B
97.8%
6
LiveCodeBench
AA archived
LiveCodeBench · AA archived: AA evaluates solutions to newly collected competitive-programming problems
AA
Gemini 3 Flash Preview
90.8%
153
+1 versions & slices
official artifact (through April 2025)
LiveCodeBench · official artifact (through April 2025): Mean pass@1 over each model’s available problems in the current official public generation artifact; its latest problem is dated April 7, 2025
LiveCodeBench
o3
84.7%
10
AIME
2025 · AA archived
AIME 2025 · AA archived: AA evaluates the 30 integer-answer problems from the 2025 AIME mathematics competition
AA
GPT-5.2
99%
140
OTIS Mock AIME 2024
2025
OTIS Mock AIME 2024 2025: Models solve the stated OTIS mock AIME mathematics problems
Epoch
Claude Fable 5
100%
130
+1 versions & slices
2025 · February 2025 run
OTIS Mock AIME 2024 2025 · Epoch independent run · historical run: Models solve the stated OTIS mock AIME mathematics problems
Epoch
DeepSeek V3
15.8%
1
BFCL
v4
BFCL v4: Tests function calling, multi-turn interactions, search and memory in the v4 suite
Berkeley
Claude Opus 4.5
77.5%
39
+2 versions & slices
v4 · prompt mode
BFCL v4 · BFCL v4 source function-call format · historical run: Tests function calling, multi-turn interactions, search and memory in the v4 suite
Berkeley
Gemini 2.5 Flash
50.9%
16
v4 · native function calls
BFCL v4 · BFCL v4 source function-call format / FC · historical run: Tests function calling, multi-turn interactions, search and memory in the v4 suite
Berkeley
o3 (2025-04-16)
48.6%
2
MMLU-Pro
official board
MMLU-Pro · official board: Graduate-level multiple-choice knowledge and reasoning across 14 subjects, including science, mathematics, law and health
Gemini 3.1 Pro
91.2%
32
SWE-bench Verified
Epoch
Swe Bench Verified: Agents modify real repositories to resolve the stated Verified issues
Epoch
Claude Opus 4.7
83.5%
24
SWE-bench Pro (public)
SWE-bench Pro (public): Agents resolve long-horizon professional repository tasks; public and private splits are separate
Muse Spark 1.1
61.5%
19
Video-MME
2 · without subtitles/audio
Video-MME 2 · without subtitles/audio: Grouped non-linear question scoring for video understanding; this is distinct from mean question accuracy
Video-MME
Gemini 3 Flash Preview
32.9%
18
+6 versions & slices
2 · with subtitles/audio
Video-MME 2 · with subtitles/audio: Grouped non-linear question scoring for video understanding; this is distinct from mean question accuracy
Video-MME
Gemini 3 Flash Preview
42.5%
18
2 · without subtitles/audio · 512 frames
Video-MME 2 · without subtitles/audio · no-subs; frames=512 · historical run: Grouped non-linear question scoring for video understanding; this is distinct from mean question accuracy
Video-MME
Kimi K2.5
33%
2
1 · without subtitles/audio
Video-MME 1 · without subtitles/audio: Question accuracy across short, medium and long videos
Video-MME
r
Aria
67.6%
1
1 · with subtitles/audio
Video-MME 1 · with subtitles/audio: Question accuracy across short, medium and long videos
Video-MME
r
Aria
72.1%
1
2 · with subtitles/audio · 512 frames
Video-MME 2 · with subtitles/audio · with-subs; frames=512 · historical run: Grouped non-linear question scoring for video understanding; this is distinct from mean question accuracy
Video-MME
Kimi K2.5
40.7%
1
2 · with subtitles/audio · native audio · 512 frames
Video-MME 2 · with subtitles/audio · native audio; frames=512 · historical run: Grouped non-linear question scoring for video understanding; this is distinct from mean question accuracy
Video-MME
Qwen3.5 Omni Plus
27.4%
1
Aider Polyglot
two attempts
Aider Polyglot · two attempts: Models edit code for 225 Exercism exercises in six languages, with a second attempt after test feedback
Aider
GPT-5
88%
15
+1 versions & slices
two attempts · diff edits · December 2024 run
Aider Polyglot · two attempts · two attempts / diff · historical run: Models edit code for 225 Exercism exercises in six languages, with a second attempt after test feedback
Aider
Qwen2.5 Coder 32B Instruct
8%
1
SWE-rebench
all · 2026-02-01 to 2026-07-01
SWE-rebench · all · 2026-02-01 to 2026-07-01: Repository issue resolution over the exact fresh pull-request window, under the source agent version
Nebius
Qwen 3.5 35B A3B
29.5%
1
+5 versions & slices
Go · 2026-02-01 to 2026-07-01
SWE-rebench · go · 2026-02-01 to 2026-07-01: Repository issue resolution over the exact fresh pull-request window, under the source agent version
Nebius
Qwen 3.5 35B A3B
18.1%
1
Java · 2026-02-01 to 2026-07-01
SWE-rebench · java · 2026-02-01 to 2026-07-01: Repository issue resolution over the exact fresh pull-request window, under the source agent version
Nebius
Qwen 3.5 35B A3B
16.2%
1
Python · 2026-02-01 to 2026-07-01
SWE-rebench · python · 2026-02-01 to 2026-07-01: Repository issue resolution over the exact fresh pull-request window, under the source agent version
Nebius
Qwen 3.5 35B A3B
42.6%
1
Rust · 2026-02-01 to 2026-07-01
SWE-rebench · rust · 2026-02-01 to 2026-07-01: Repository issue resolution over the exact fresh pull-request window, under the source agent version
Nebius
Qwen 3.5 35B A3B
22.4%
1
TypeScript · 2026-02-01 to 2026-07-01
SWE-rebench · typescript · 2026-02-01 to 2026-07-01: Repository issue resolution over the exact fresh pull-request window, under the source agent version
Nebius
Qwen 3.5 35B A3B
16.7%
1
No benchmark matches
Model
Humanity's Last Exam
Per task
Price
Speed
Supports
Context
Mercury 2
inception
17.1
high
$0.039
$0.25
/
$0.75
487
128K
Mercury 2.5
inception
11.8
thinking
$0.12
$0.20
/
$0.75
781
260K
Show 2 not measured on Humanity's Last Exam
