Benchmarks › AutomationBench-AA

AutomationBench-AA

By Zapier (AutomationBench); Artificial Analysis (AA evaluation) ↗Paper ↗
Complete business workflows by calling app APIs
77.5top score · Gemini 4 Argon
133models ranked
80open-weights models
Guardrail-gated objective completionhigher is better
Oct 20, 2026last updated

How the test works

1

Task

657 tasks

  • What the model doesComplete business workflows by calling app APIs
  • Kinds of tasksSaaS workflow automation
  • DomainsFinance, HR, Marketing, Operations, Sales, Support
2

Run

1 run per task

  • Step limit50 steps
  • Token budgetNon-reasoning: up to 16,384 output tokens, adjusted to model limits; reasoning: model-specific maximum
  • Temperature0 for non-reasoning; 0.6 for reasoning, unless the model lab recommends otherwise
3

Score

Guardrail-gated objective completion

  • Graded byProgrammatic final-state checks; any guardrail violation or error scores zero
  • Better isHigher
  • Test setPrivate held-out split
Sources: artificialanalysis.ai

AutomationBench-AA leaderboardtop 10 of 133

Full leaderboard with filters →
ModelAutomationBench-AAPer taskPriceSpeedSupportsContext
Gemini 4 ArgonGoogle
77.5high
$1.99$2 / $10841M
MiniMax M3.1 FlashMiniMax
72.5thinking
$0.22$0.15 / $0.601051M
Claude Sonnet 5.5Anthropic
71.8max
$5.46$2 / $101371M
Muse Wonder FullMeta
71.3
——1M
Claude Opus 5.5Anthropic
69.5max
$5.98$4 / $20961M
GPT-6 AstraOpenAI
68.5max
$3.26$10 / $50691.05M
Muse WonderMeta
68.1max
$4.52$4 / $201611M
Grok 4.6xAI
67xhigh
$1.75$2 / $665500K
DeepSeek V4.1 Flash 1017DeepSeekOpen
66.6max
$0.26$0.30 / $1.201671M
GPT-6.1 SolOpenAI
66.6xhigh
$0.39$2 / $10531.05M

Versions and boards on Model Menu

BoardModels
AutomationBench-AAShown above
133
AutomationBench · Zapier run
2

AutomationBench-AA questions

What is AutomationBench-AA?›

Complete business workflows by calling app APIs. It has 657 tasks. Created by Zapier (AutomationBench); Artificial Analysis (AA evaluation).

Which model is best on AutomationBench-AA?›

Gemini 4 Argon leads AutomationBench-AA with 77.5, ahead of MiniMax M3.1 Flash (72.5) and Claude Sonnet 5.5 (71.8).

What is the best open-weights model on AutomationBench-AA?›

DeepSeek V4.1 Flash 1017 with 66.6, then MiMo v2.6 Flash (64.1).

How is AutomationBench-AA scored?›

Guardrail-gated objective completion; programmatic final-state checks; any guardrail violation or error scores zero; 1 run per task per task.

Which versions of AutomationBench-AA are there?›

AutomationBench-AA (133 models), AutomationBench · Zapier run (2 models).