simple-bench.com

SimpleBench is a multiple-choice text benchmark for large language models built around everyday human reasoning rather than memorized knowledge. Its headline result is that a non-specialized human baseline of 83.7% — drawn from a small sample of nine participants — outperforms every tested LLM, including the top model listed, Claude Fable, at 81.9%. If you want to know whether a frontier model can handle basic reasoning that a high-school-level person manages, SimpleBench is designed to test exactly that gap.

What SimpleBench measures

The benchmark contains over 200 questions across three areas:

• Spatio-temporal reasoning — reasoning about space, time, and how objects or events relate.
• Social intelligence — questions that depend on understanding people, intentions, or social context.
• Linguistic adversarial robustness — what the project calls "trick questions," where wording is designed to mislead a model that pattern-matches instead of reasoning.

The framing matters: most text-based benchmarks show LLMs beating non-specialized humans and increasingly exceeding expert human performance. SimpleBench is presented as a case where that pattern breaks down, which the team attributes to frontier models relying on memorized knowledge and approximate reasoning retrieval rather than genuine reasoning.

How to read the leaderboard

The leaderboard reports several fields you need to interpret together:

Two footnotes on the leaderboard are worth noting. The human baseline calculation is detailed in the Human Evaluation section of the report, and the team also tried an engineered prompt to optimize benchmark-specific performance, described in the LLM Eval section. The leaderboard also offers views for MCQ, Perf/$, Timeline, and Open-Ended, so you can compare raw accuracy, cost-efficiency, results over time, or open-ended performance depending on what you care about.

Because scores are averaged over 5 runs, small differences between adjacent models should be read with that variance in mind rather than as a definitive ordering.

Where to go deeper

The page links to the public dataset, the code, and the full technical report, plus a "Try Yourself" option if you want to attempt the questions directly. You can also use the listed models through the Simple Bench app on LMcouncil.ai. For the latest fully explained update, the project points to its Patreon, and inquiries go to [email protected].

The practical takeaway: use SimpleBench when you want a reasoning-focused counterweight to knowledge-heavy benchmarks, and read the score alongside the settings and the human baseline rather than in isolation.

A large language model (LLM) is a neural network trained to predict the next token in a sequence, and it generates text by repeating that prediction one token at a time. Under the hood, the text you type is split into tokens, converted into vectors (embeddings), and passed through a stack of transformer layers where an attention mechanism lets each token weigh the others. The final layer scores every possible next token, and one is selected. This explanation fits anyone who wants a working mental model of the pipeline rather than a math-heavy derivation; AnimatedLLM (animatedllm.github.io) is built specifically to visualize these internal steps.

The core components

The vocabulary is fixed, so every input and output is expressed in terms of tokens the model already knows.

From text to prediction, step by step

1. Tokenize. Input text is split into tokens. A word may become one token or several subword pieces.
2. Embed. Each token maps to a vector. Position information is added so the model knows order.
3. Pass through transformer layers. Each layer applies attention (tokens exchange information) and a feed-forward network (each position is transformed).
4. Produce logits. The final representation is projected onto the vocabulary, giving a score for every possible next token.
5. Select a token. The highest score wins under greedy decoding; sampling methods can pick lower-ranked tokens for variety.
6. Append and repeat. The chosen token is added to the sequence, and the loop runs again to produce the next one.

The expected result at each loop is a single new token; a full response is just this loop repeated.

How attention works, in plain terms

Attention answers: "for this token, which other tokens matter right now?" Each token produces a query, and every token produces a key and a value. The query is compared against all keys to get weights, and the values are combined using those weights. So a pronoun can gather information from the noun it refers to, and a verb can look back at its subject.

This is why context length matters: attention can only weigh tokens inside the window it is given. Multi-head attention runs several of these weightings in parallel so different relationships can be tracked at once.

Training vs. inference

These are the same architecture used in two different modes:

• Training: the model sees text with the next token known, predicts it, measures the error, and adjusts weights across many examples. This is where knowledge and language patterns are learned.
• Inference: weights are frozen. The model only predicts forward, token by token, with no learning. This is what happens when you use a chat model.

A useful intuition: training is studying with an answer key; inference is answering without one.

Building intuition with interactive visualizations

Reading the steps is not the same as seeing them. AnimatedLLM is an interactive resource whose stated purpose is to help you understand how large language models work under the hood. Use it to watch tokens become vectors, follow attention weights between positions, and trace how a prediction is produced. For example, if you want to see why a model completes "The cat sat on the ___" with "mat," step through the attention view to observe which earlier tokens the final position is weighting most heavily.

Common sticking points

• Tokens are not words. A single word can be multiple tokens, which is why models sometimes miscount letters.
• Embeddings are not meanings you can read off. They are learned coordinates; similarity is geometric, not definitional.
• Attention is not memory. It operates within the current context window, not a persistent store.
• One token at a time. Fluency comes from repetition, not from generating a whole sentence in one pass.

If you want to go from "I've heard of transformers" to "I can trace the pipeline," work through the tokenize → embed → attend → score → select loop once, then use an interactive demo to watch each stage on real text.

The o1 series is OpenAI's family of reasoning models, and SimpleBench treats it as an exception to its standard sampling configuration: the benchmark runs at temperature 0.7 and top-p 0.95 for all models except o1 series. That exception exists because reasoning models generate an internal chain of thought before answering, so the sampling parameters that shape a normal single-pass response don't apply the same way. If you're comparing o1 results against other models on SimpleBench, the score is still directly comparable — only the generation settings differ.

What SimpleBench measures

SimpleBench is a multiple-choice text benchmark built to test everyday reasoning rather than specialized knowledge. According to the SimpleBench team, it contains over 200 questions spanning:

• Spatio-temporal reasoning — tracking objects, positions, and sequences
• Social intelligence — reasoning about people, intentions, and situations
• Linguistic adversarial robustness — "trick questions" designed to catch pattern-matching rather than reasoning

The headline finding is that a non-specialized human baseline scores 83.7% (based on a small sample of nine participants), outperforming every tested LLM. The top model listed, Claude Fable, scored 81.9%. The team's conclusion is that memorized knowledge and approximate reasoning retrieval are not always enough to answer basic questions.

Why the o1 series is flagged as an exception

The leaderboard's benchmark settings note reads: temperature: 0.7, top-p: 0.95 (except o1 series).

Temperature and top-p are sampling controls. Temperature scales how randomly the model picks among likely next tokens; top-p (nucleus sampling) restricts selection to the smallest set of tokens whose cumulative probability crosses a threshold. Both assume the model produces its answer in one forward pass of token-by-token generation.

Reasoning models like o1 work differently: they spend additional computation generating a reasoning trace before producing a final answer. That changes what the sampling settings are actually controlling, so applying the same values used for standard models would not be a like-for-like configuration. SimpleBench's note is a disclosure of that difference, not a claim that o1 was given an advantage or handicap.

How this affects reading the leaderboard

Two further caveats apply to any model on this leaderboard:

• The human baseline comes from a small sample of nine participants, so treat 83.7% as an estimate, not a precise population figure.
• SimpleBench also reports trying an engineered prompt to optimize benchmark-specific performance for some models. That means a listed score may reflect prompt tuning, not just raw model capability.

What the results imply about o1-style reasoning

SimpleBench's own framing is that frontier LLMs rely on memorized knowledge and approximate reasoning retrieval, and that this is "not always enough to answer basic questions just yet." The gap between the 83.7% human baseline and the best model score of 81.9% is small, but it sits on questions that require no specialized training — only everyday reasoning.

For o1 specifically, the benchmark's design means its score reflects performance on trick questions and spatio-temporal/social reasoning under a reasoning-model generation setup. It does not tell you how o1 performs on knowledge-heavy tasks, coding, or math, which SimpleBench does not test.

Practical takeaways

1. Don't read the settings exception as a scoring adjustment. It's a transparency note about how reasoning models are sampled.
2. Compare scores, not settings. The leaderboard is designed so all models face the same questions.
3. Check the report for methodology. SimpleBench points to its full technical report for the human evaluation section and the LLM eval section covering the engineered prompt.
4. Try the questions yourself. SimpleBench offers a public dataset and a "Try Yourself" option, plus the models via the Simple Bench app on LMcouncil.ai — useful if you want to see whether the trick questions fool you too.

A large language model (LLM) is a neural network trained on massive amounts of text to predict the next token in a sequence; that single training objective, scaled up, produces the fluent question-answering behavior you see in chatbots. The catch is that this objective rewards plausible continuation, not verified reasoning — which is why on SimpleBench, a non-specialized human baseline of 83.7% outperforms every tested LLM, including the top model at 81.9%. If you want to understand where LLMs are strong (recall, fluency, pattern completion) versus where they break (multi-step logic, social inference, trick questions), the sections below explain the mechanism and give you a way to test it yourself.

The core definition, in plain terms

An LLM is a neural network with a very large number of parameters, trained on a broad corpus of text. During training it repeatedly sees a chunk of text and is asked: what token comes next? Every weight update is driven by how wrong that prediction was.

Two consequences follow directly from this:

• Knowledge is compressed into weights. The model does not look anything up in a database at inference time; it reconstructs an answer from statistical patterns baked into its parameters.
• Generation is sequential. The output is built one token at a time, and each new token is conditioned on everything before it — including tokens the model itself just produced.

That second point matters for reasoning: an early wrong step becomes part of the context the model conditions on, so errors can compound rather than self-correct.

From pretraining to a usable assistant

The typical pipeline has a few recognizable stages:

1. Pretraining — next-token prediction over a large text corpus. This is where most knowledge and linguistic competence come from.
2. Instruction tuning — training on examples of instructions paired with good responses, so the model answers requests instead of just continuing text.
3. Alignment — further tuning (often with human or model feedback) to make outputs helpful, harmless, and appropriately formatted.

Parameter scale correlates with capability, but not linearly and not on every task. A model can memorize enormous amounts of factual content and still fail a question that requires holding several constraints in mind at once — which is exactly the gap SimpleBench is designed to expose.

Why "reasoning" here is closer to retrieval than deduction

When an LLM answers a reasoning question, it is not running a formal proof. It is producing the continuation that most resembles reasoning text in its training data. For many questions this approximation lands on the right answer, because the pattern is common. It fails when:

• The question is novel in structure and no close pattern exists in training.
• The question is adversarial — phrased so the statistically likely continuation is the wrong answer (a trick question).
• The answer requires chaining several steps where each step must be exactly right.
• The answer depends on social or spatial inference that humans do implicitly but that is rarely spelled out in text.

SimpleBench targets precisely these categories: spatio-temporal reasoning, social intelligence, and linguistic adversarial robustness. Its 200+ multiple-choice questions are built so that unspecialized (high-school-level) knowledge is sufficient — the difficulty is in the reasoning, not the facts.

What SimpleBench actually shows

The headline finding is not that models are weak overall — on most text benchmarks LLMs now beat non-specialized humans and increasingly approach expert level. The finding is narrower and more useful: memorized knowledge plus approximate reasoning retrieval is not always enough for basic questions. The human advantage on SimpleBench comes from the kind of reasoning that does not require specialized training but does require actually working through the problem.

Note the caveats the source itself flags: the human baseline rests on a small sample of nine participants, and the benchmark applies an engineered prompt to optimize benchmark-specific performance for models. Treat the 83.7% vs. 81.9% gap as directional evidence about failure modes, not as a precise ranking.

How to verify this yourself

You do not need the full benchmark to see the pattern. Try this:

1. Pick a question with a known answer that requires 3+ dependent steps — for example, a scheduling puzzle where each constraint eliminates options. Write down the correct answer first.
2. Ask the model the question as plain text, with no hints about the method.
3. Compare the model's final answer to yours, then read its intermediate steps. The interesting failure is not a wrong final answer — it is a step where the model asserts something that does not follow, then builds on it.
4. Rephrase the same question as a trick version where the obvious reading is wrong. Models that handled the straightforward version often fail here, because the likely continuation is the trap.
5. Repeat with temperature low to separate sampling randomness from a genuine reasoning gap.

What you are looking for is a consistent pattern: correct on familiar phrasings, wrong on novel or adversarial ones. That pattern is the practical signature of next-token prediction standing in for reasoning.

What this means for choosing or using a model

• For recall-heavy tasks (summarizing, drafting, retrieving known facts), the pretraining-plus-tuning pipeline is well matched, and benchmark gaps like SimpleBench's matter little.
• For multi-step or adversarial reasoning, treat the model's output as a draft to check, not a conclusion. The SimpleBench result is direct evidence that a capable model can lose to an ordinary person on questions requiring no special knowledge.
• When you evaluate a model, prefer tasks with dependent steps and counterintuitive phrasings over fact-recall questions — the latter will make almost any frontier model look strong and tell you nothing about its reasoning limits.

The short version: an LLM is a next-token predictor whose fluency makes it look like a reasoner. SimpleBench is useful because it isolates the cases where that appearance breaks down.

A reasoning LLM is a large language model that is trained or prompted to spend extra computation on a problem before answering — producing an explicit chain of thought, searching over candidate steps, or verifying its own output — instead of emitting the most likely next token immediately. The distinction matters when a task requires multi-step logic (math, code, planning) rather than fluent recall. It matters less for summarization, translation, or simple retrieval, where a standard LLM is usually faster and cheaper.

The core difference: next-token prediction vs. deliberate computation

A standard LLM is, at its base, a next-token predictor. Given a prompt, it samples the most probable continuation. That works surprisingly well for language tasks, but it means the model commits to an answer as it generates, with no built-in step to check whether the reasoning holds.

A reasoning LLM adds a deliberate phase before the final answer. Depending on the model, that phase can involve:

• Longer chains of thought — the model writes out intermediate steps rather than jumping to a conclusion.
• Search over alternatives — exploring more than one path and comparing them.
• Self-verification — checking a draft answer against the problem before presenting it.
• Reinforcement learning on reasoning traces — training the model to produce chains that lead to correct answers, not just plausible-sounding ones.

The practical consequence is that reasoning models trade latency and token cost for accuracy on problems where a single forward pass tends to fail.

How to tell whether a model is a reasoning model

You do not need internal access to the weights. Look for these observable signals:

SimpleBench's leaderboard notes one concrete example of that last point: its benchmark settings use temperature 0.7 and top-p 0.95, except for the o1 series. That exception exists because reasoning models behave differently under standard sampling settings — a useful tell that a model family is being treated as a distinct category.

How reasoning ability gets measured: the SimpleBench case

SimpleBench is a multiple-choice text benchmark built specifically to probe everyday reasoning rather than specialized knowledge. According to the SimpleBench page, it contains over 200 questions covering three areas:

• Spatio-temporal reasoning — reasoning about space, time, and physical situations.
• Social intelligence — reasoning about people, intentions, and social context.
• Linguistic adversarial robustness — "trick questions" designed to catch models that pattern-match instead of reading carefully.

The design premise is stated directly on the page: on most text benchmarks, LLMs now beat non-specialized humans and increasingly beat experts. SimpleBench inverts that. Its non-specialized (high school-level) human baseline is 83.7%, based on a small sample of nine participants, and that baseline outperforms every tested LLM — including the top model listed, Claude Fable, at 81.9%.

That gap is the interesting part for anyone evaluating a reasoning LLM. If frontier models with extensive reasoning training still fall short of a nine-person high-school-knowledge baseline on basic questions, then memorized knowledge and approximate reasoning retrieval are not sufficient on their own. The SimpleBench authors frame the result exactly that way: the results "confirm that the memorized knowledge, and approximate reasoning retrieval, utilized by frontier LLMs is not always enough to answer basic questions just yet."

Two caveats worth holding onto when reading that number:

1. The human baseline comes from a small sample of nine participants, so treat 83.7% as an estimate, not a precise population figure.
2. The page notes that an engineered prompt was tried to optimize benchmark-specific performance, so leaderboard scores reflect some prompt tuning rather than raw default behavior.

What this means for choosing a model

The SimpleBench result is a useful corrective to the assumption that "reasoning model" automatically means "better at reasoning." It suggests the category is real but incomplete — reasoning training improves multi-step performance without closing the gap to ordinary human judgment on everyday problems.

A practical way to decide:

• Use a reasoning LLM when the task has verifiable intermediate steps and errors compound — multi-step math, code generation and debugging, planning with constraints, or analysis where you want the model to show its work.
• Use a standard LLM when the task is generation or transformation without deep logic — drafting, summarizing, translating, reformatting, or simple Q&A — where the extra thinking budget adds cost and latency without improving the output.
• Do not assume a reasoning model wins on trick questions or social reasoning. SimpleBench's categories are precisely the ones where the tested frontier models still trail a non-specialized human baseline.

If you want to check current standings yourself, the SimpleBench page provides a leaderboard, a public dataset, and a "Try Yourself" option, plus a full technical report for methodology details.