What benchmarks and scoring dimensions does AudioAgentBench use?

AudioAgentBench is the benchmark suite on AudioArena that evaluates voice agents through static multi-turn conversations. It contains six named benchmarks — Appointment, Assistant, Conversation, Event, Grocery, and Product — and each one is scored across five dimensions. You can select a benchmark, a model, and a run to read turn-by-turn transcripts, and a separate methodology page explains how the scoring works.

The six benchmarks

Each benchmark is a self-contained scenario type rather than a single prompt, and each is multi-turn, so a model is judged on how a conversation develops rather than on one isolated reply.

The names map to everyday voice-agent jobs: booking something, handling a general request, holding a conversation, organizing around an event, assembling a grocery order, or dealing with a product. Because all six are static multi-turn benchmarks, the same conversation is replayed to each model, which is what makes cross-model comparison meaningful.

The five scoring dimensions

Every benchmark is scored across five dimensions. The site states the count and points to a methodology page for the details of how scoring works; the specific dimension names and their definitions live on that methodology page rather than in the benchmark listing itself.

Practical consequence: when you compare two models, don't collapse the five dimensions into one number. A model can be strong on some dimensions and weak on others within the same benchmark, and the turn-by-turn transcripts are where you see why.

How to read a result

1. Pick a benchmark that matches the job you care about — for example, Grocery if you're evaluating order-taking.
2. Pick the models you want to compare.
3. Pick a run and open the turn-by-turn transcript.
4. Read the transcript alongside the dimension scores, so a low score is tied to a specific turn rather than left as an abstract rating.

Expected result: you get a per-benchmark, per-model view of a full conversation, not a single headline score.

Common sticking points

• Benchmark name ≠ difficulty level. Appointment and Grocery are different task types, not easier or harder versions of each other. Choose by task fit.
• Static multi-turn means fixed scripts. Results reflect performance on those scripted conversations; they don't automatically generalize to unscripted live traffic.
• Dimension definitions matter. Before drawing conclusions from a score, read the methodology page so you know what each dimension is actually measuring.
• Transcripts are the evidence. If two models score similarly, the turn-by-turn transcripts are what tell you whether the difference matters for your use case.

AudioAgentBench is published by Arcada Labs, with a GitHub presence and attribution to The Intelligence Company.