简体中文
EnglishWhat the TERMINAL-BENCH Leaderboard Metrics Mean
Updated
The TERMINAL-BENCH leaderboard reports six columns — Rank, Model, Agent, Resolution Rate, Cost, and Tokens — and the headline number is the resolution rate: the share of Terminal-Bench 4.0 tasks an agent completes successfully. The whiskers around each resolution rate span the 95% confidence interval, so two entries whose intervals overlap are not clearly separated by the data. Use resolution rate to judge capability, and Cost/Tokens to judge what that capability costs to run.
The columns, one by one
| Column | What it tells you |
|---|---|
| Rank | Ordering of entries on the leaderboard, driven by resolution rate. |
| Model | The underlying model being evaluated. |
| Agent | The agent harness or scaffold wrapped around the model. |
| Resolution Rate | Fraction of Terminal-Bench 4.0 tasks the agent resolved. |
| Cost | The cost recorded for the run. |
| Tokens | Token usage recorded for the run. |
The split between Model and Agent matters: the same model can appear more than once with different agents, and the resolution rate belongs to that model-plus-agent combination, not to the model alone.
Reading the resolution rate
The resolution rate is the primary signal. A higher value means the agent resolved a larger share of Terminal-Bench 4.0 tasks. Because it is a rate over a task set, it is comparable across entries on the same benchmark version — but only within that version. Terminal-Bench 4.0 results should not be compared against results from other benchmark versions.
Reading the 95% confidence interval (whiskers)
The whiskers are not decoration. They express uncertainty in the measured rate:
- A narrow interval means the estimate is precise.
- A wide interval means the estimate is loose — often because fewer tasks or runs back it.
- If two entries' intervals overlap, the leaderboard ordering between them is not statistically clear-cut; treat them as roughly tied rather than strictly ranked.
A common misread is to treat Rank as a strict ordering. Rank reflects the point estimate; the whiskers tell you how much confidence that ordering deserves.
Comparing models and agents with these metrics
- Compare resolution rates first, and check whether the confidence intervals overlap before declaring a winner.
- Then compare Cost and Tokens for entries whose resolution rates are close. A marginally higher resolution rate at a much higher cost is a different trade-off than a clear capability gap.
- Hold the benchmark version constant. Only Terminal-Bench 4.0 numbers belong in a Terminal-Bench 4.0 comparison.
- Compare like with like. Since resolution rate is tied to a model-agent pair, compare specific pairings rather than assuming a model's best agent result transfers to another harness.
What the metrics do not tell you
The leaderboard reports aggregate rates, cost, and tokens. It does not break down which task categories an agent handles well, and it does not establish that a higher-ranked entry will perform better on your own workload. Treat the leaderboard as a capability-and-cost signal for Terminal-Bench 4.0 tasks, and validate on your own tasks before committing.