Benchmarks › ITBench
ITBench
Diagnose root causes from offline Kubernetes incident data
56.2top score · GPT-5.6 Sol
37models ranked
17open-weights models
Precision at full recallhigher is better
Oct 20, 2026last updated
How the test works
1
Task
59 tasks
- What the model doesDiagnose root causes from offline Kubernetes incident data
- Kinds of tasksSite reliability incident diagnosis
- In → outText, incident data → JSON diagnosis
2
Run
3 runs per task
- Token budgetNon-reasoning: up to 16,384 output tokens, adjusted to model limits; reasoning: model-specific maximum
- Temperature0 for non-reasoning; 0.6 for reasoning, unless the model lab recommends otherwise
3
Score
Precision at full recall
- Graded byLLM-normalized entity matching; zero if any root cause is missed
- Better isHigher
Sources: artificialanalysis.ai
ITBench SRE leaderboardtop 10 of 37
Full leaderboard with filters →| Model | ITBench SRE | Per task | Price | Speed | Supports | Context |
|---|---|---|---|---|---|---|
56.2max | $1.99 | $4 / $20 | 114 | 1.05M | ||
55.6thinking | $1.03 | $1 / $2.70 | 93 | 1M | ||
52.5high | $1.24 | $0.75 / $3.75 | 149 | 1M | ||
51.2max | $0.28 | $0.15 / $0.50 | 179 | 1M | ||
51max | $1.40 | $2 / $12 | 142 | 1.05M | ||
49.5max | $7.63 | $10 / $50 | 69 | 1M | ||
49.4max | $1.04 | $2 / $10 | 128 | 1.05M | ||
48.6max | $3.26 | $10 / $50 | 69 | 1.05M | ||
47.7max | $2.00 | $3 / $15 | 112 | 1M | ||
46.7max | — | $5 / $25 | 63 | 1M |
ITBench questions
What is ITBench?›
Diagnose root causes from offline Kubernetes incident data. It has 59 tasks. Created by IBM Research (ITBench-AA with Artificial Analysis).
Which model is best on ITBench?›
GPT-5.6 Sol leads ITBench SRE with 56.2, ahead of Step 5 Preview (55.6) and Gemini 3.8 Flash (52.5).
What is the best open-weights model on ITBench?›
Step 5 Preview with 55.6, then GLM 5.3 Flash (51.2).
How is ITBench scored?›
Precision at full recall; llm-normalized entity matching; zero if any root cause is missed; 3 runs per task per task.