What Is SimpleBench and How Does It Measure LLM Reasoning?

SimpleBench is a multiple-choice text benchmark designed to test whether large language models can handle everyday reasoning tasks that a person with ordinary high-school-level knowledge can solve. Its headline finding is that a non-specialized human baseline of 83.7% (from a small sample of nine participants) still beats every tested LLM, including the top model listed, Claude Fable, at 81.9%. Use it if you want a reasoning-focused evaluation that deliberately separates general reasoning from specialized domain knowledge; don't treat it as a general-purpose capability score.

What SimpleBench actually measures

The benchmark targets three reasoning categories:

• Spatio-temporal reasoning — questions about objects, positions, movement, and ordering in space and time.
• Social intelligence — questions that require reasoning about people, intentions, or social situations.
• Linguistic adversarial robustness — what the site calls "trick questions," where surface wording can mislead a model away from the correct answer.

The design intent is to probe reasoning that doesn't depend on memorized facts or expert knowledge. According to the site, the results "confirm that the memorized knowledge, and approximate reasoning retrieval, utilized by frontier LLMs is not always enough to answer basic questions just yet."

How the benchmark is structured

The human baseline comes from a small sample, so treat the exact gap as indicative rather than precise. The site notes that details on how the baseline was calculated are in the Human Evaluation section of its report.

How to read the leaderboard

The leaderboard reports a Score (AVG@5) — an average over five runs — along with organization, and offers views for MCQ, Perf/$, Timeline, and Open-Ended. Benchmark settings are:

• temperature: 0.7
• top-p: 0.95
• Exception: the o1 series uses different settings.

Two footnotes matter for interpretation:

• The human baseline calculation is documented in the report's Human Evaluation section.
• For some entries, the team tried an engineered prompt to optimize benchmark-specific performance, described in the report's LLM Eval section.

That second point means a leaderboard score may reflect prompt tuning for this benchmark specifically, not just raw model ability. When comparing models, check whether an entry used an engineered prompt.

How to try it yourself

The site provides three entry points: a public dataset, code, and the latest leaderboard report. It also points to the Simple Bench app at LMcouncil.ai for using the models. If you want to reproduce or extend results, start from the public dataset and code rather than the leaderboard alone, since the leaderboard aggregates runs under fixed settings.

What SimpleBench does and doesn't tell you

It does tell you: whether a model can handle short, everyday reasoning questions where a non-specialized human succeeds, and how models compare under a fixed multiple-choice protocol.

It doesn't tell you: overall model quality, performance on expert-level or open-ended tasks, or how a model behaves outside the benchmark's temperature/top-p settings. The gap between the human baseline and the best model is the point — frontier LLMs can still miss basic reasoning questions despite strong performance on most text benchmarks.

For the full methodology and current numbers, consult the technical report and leaderboard on the SimpleBench site, since scores and model rankings change over time.