Benchmarks › MMLU

MMLU

Older benchmarkBy UC Berkeley, Columbia University, University of Chicago and UIUC ↗Results by TIGER-Lab ↗Since 2024Paper ↗
Answer harder academic questions with ten answer choices
91.2top score · Gemini 3.1 Pro
32models ranked
24open-weights models
Overall accuracyhigher is better
Oct 20, 2026last updated

How the test works

1

Task

12,032 questions in the paper suite

  • What the model doesAnswer harder academic questions with ten answer choices
  • Kinds of tasksTen-option multiple-choice questions
  • DomainsBiology, Business, Chemistry, Computer Science, Economics, Engineering, Health, History, Law, Math, Philosophy, Physics, Psychology, Others
2

Run

Official MMLU-Pro 5-shot chain-of-thought reference protocol

  • How runs combineMicro accuracy: sum correct across subjects divided by sum correct plus wrong; not an unweighted subject mean.
3

Score

Overall accuracy

  • Graded byExtract A–J using regex fallbacks and compare with the answer key; no LLM judge
  • Better isHigher
Sources: github.com, arxiv.org

MMLU-Pro · Official board leaderboardtop 10 of 32

Full leaderboard with filters →
ModelMMLU-Pro · Official boardPer taskPriceSpeedSupportsContext
Gemini 3.1 ProGoogle
91.2unstated
—$2 / $12—1M
GPT-4oOpenAI
74.7unstated
—$2.50 / $10—128K
GPT OSS 20BOpenAIOpen
73.1medium
—$0.018 / $0.09—131K
QwQ 32BAlibaba (Qwen)Open
69.1
—$0.15 / $0.40—131K
Llama 3.3 70B InstructMetaOpen
65.9unstated
—$0.10 / $0.32—128K
GPT-4o MiniOpenAI
63.1unstated
—$0.15 / $0.60—128K
Llama 3.1 70B InstructMetaOpen
62.8unstated
—$0.40 / $0.40—131K
mReflection Llama 3.1 70BmattshumerOpen
60.4
—$0.35 / $0.40—8K
LEXAONE 3.5 32B InstructLGAI-EXAONEOpen
58.9
——33K
Gemma 2 27BGoogleOpen
56.5unstated
—$0.27 / $0.27—8K

MMLU questions

What is MMLU?›

Answer harder academic questions with ten answer choices. It has 12,032 questions in the paper suite. Created by UC Berkeley, Columbia University, University of Chicago and UIUC.

Which model is best on MMLU?›

Gemini 3.1 Pro leads MMLU-Pro · Official board with 91.2, ahead of GPT-4o (74.7) and GPT OSS 20B (73.1).

What is the best open-weights model on MMLU?›

GPT OSS 20B with 73.1, then QwQ 32B (69.1).

How is MMLU scored?›

Overall accuracy; extract a–j using regex fallbacks and compare with the answer key; no llm judge.