What Is SimpleBench and What Does Its LLM Leaderboard Measure?

Updated

SimpleBench is a multiple-choice text benchmark for large language models built around everyday human reasoning rather than memorized knowledge. Its headline result is that a non-specialized human baseline of 83.7% — drawn from a small sample of nine participants — outperforms every tested LLM, including the top model listed, Claude Fable, at 81.9%. If you want to know whether a frontier model can handle basic reasoning that a high-school-level person manages, SimpleBench is designed to test exactly that gap.

What SimpleBench measures

The benchmark contains over 200 questions across three areas:

  • Spatio-temporal reasoning — reasoning about space, time, and how objects or events relate.
  • Social intelligence — questions that depend on understanding people, intentions, or social context.
  • Linguistic adversarial robustness — what the project calls "trick questions," where wording is designed to mislead a model that pattern-matches instead of reasoning.

The framing matters: most text-based benchmarks show LLMs beating non-specialized humans and increasingly exceeding expert human performance. SimpleBench is presented as a case where that pattern breaks down, which the team attributes to frontier models relying on memorized knowledge and approximate reasoning retrieval rather than genuine reasoning.

How to read the leaderboard

The leaderboard reports several fields you need to interpret together:

Field What it tells you
Rank Position on the benchmark
Model The LLM being evaluated
Score (AVG@5) Average score across 5 runs
Organization Who produced the model
Benchmark settings Temperature 0.7, top-p 0.95 (except the o1 series)

Two footnotes on the leaderboard are worth noting. The human baseline calculation is detailed in the Human Evaluation section of the report, and the team also tried an engineered prompt to optimize benchmark-specific performance, described in the LLM Eval section. The leaderboard also offers views for MCQ, Perf/$, Timeline, and Open-Ended, so you can compare raw accuracy, cost-efficiency, results over time, or open-ended performance depending on what you care about.

Because scores are averaged over 5 runs, small differences between adjacent models should be read with that variance in mind rather than as a definitive ordering.

Where to go deeper

The page links to the public dataset, the code, and the full technical report, plus a "Try Yourself" option if you want to attempt the questions directly. You can also use the listed models through the Simple Bench app on LMcouncil.ai. For the latest fully explained update, the project points to its Patreon, and inquiries go to [email protected].

The practical takeaway: use SimpleBench when you want a reasoning-focused counterweight to knowledge-heavy benchmarks, and read the score alongside the settings and the human baseline rather than in isolation.