What Is the o1 Series in SimpleBench and Why Does It Use Different Benchmark Settings?

Updated

The o1 series is OpenAI's family of reasoning models, and SimpleBench treats it as an exception to its standard sampling configuration: the benchmark runs at temperature 0.7 and top-p 0.95 for all models except o1 series. That exception exists because reasoning models generate an internal chain of thought before answering, so the sampling parameters that shape a normal single-pass response don't apply the same way. If you're comparing o1 results against other models on SimpleBench, the score is still directly comparable — only the generation settings differ.

What SimpleBench measures

SimpleBench is a multiple-choice text benchmark built to test everyday reasoning rather than specialized knowledge. According to the SimpleBench team, it contains over 200 questions spanning:

  • Spatio-temporal reasoning — tracking objects, positions, and sequences
  • Social intelligence — reasoning about people, intentions, and situations
  • Linguistic adversarial robustness — "trick questions" designed to catch pattern-matching rather than reasoning

The headline finding is that a non-specialized human baseline scores 83.7% (based on a small sample of nine participants), outperforming every tested LLM. The top model listed, Claude Fable, scored 81.9%. The team's conclusion is that memorized knowledge and approximate reasoning retrieval are not always enough to answer basic questions.

Why the o1 series is flagged as an exception

The leaderboard's benchmark settings note reads: temperature: 0.7, top-p: 0.95 (except o1 series).

Temperature and top-p are sampling controls. Temperature scales how randomly the model picks among likely next tokens; top-p (nucleus sampling) restricts selection to the smallest set of tokens whose cumulative probability crosses a threshold. Both assume the model produces its answer in one forward pass of token-by-token generation.

Reasoning models like o1 work differently: they spend additional computation generating a reasoning trace before producing a final answer. That changes what the sampling settings are actually controlling, so applying the same values used for standard models would not be a like-for-like configuration. SimpleBench's note is a disclosure of that difference, not a claim that o1 was given an advantage or handicap.

How this affects reading the leaderboard

Question Answer
Is o1's score comparable to other models' scores? Yes — all models answer the same 200+ questions, scored the same way
Are the generation settings identical? No — o1 series uses different sampling parameters
Does SimpleBench say o1 beats the human baseline? No — the human baseline of 83.7% outperforms every tested LLM listed
Does SimpleBench claim o1 is a reasoning model? It groups o1 under the "except o1 series" settings note; the reasoning-model framing comes from OpenAI's o1 family

Two further caveats apply to any model on this leaderboard:

  • The human baseline comes from a small sample of nine participants, so treat 83.7% as an estimate, not a precise population figure.
  • SimpleBench also reports trying an engineered prompt to optimize benchmark-specific performance for some models. That means a listed score may reflect prompt tuning, not just raw model capability.

What the results imply about o1-style reasoning

SimpleBench's own framing is that frontier LLMs rely on memorized knowledge and approximate reasoning retrieval, and that this is "not always enough to answer basic questions just yet." The gap between the 83.7% human baseline and the best model score of 81.9% is small, but it sits on questions that require no specialized training — only everyday reasoning.

For o1 specifically, the benchmark's design means its score reflects performance on trick questions and spatio-temporal/social reasoning under a reasoning-model generation setup. It does not tell you how o1 performs on knowledge-heavy tasks, coding, or math, which SimpleBench does not test.

Practical takeaways

  1. Don't read the settings exception as a scoring adjustment. It's a transparency note about how reasoning models are sampled.
  2. Compare scores, not settings. The leaderboard is designed so all models face the same questions.
  3. Check the report for methodology. SimpleBench points to its full technical report for the human evaluation section and the LLM eval section covering the engineered prompt.
  4. Try the questions yourself. SimpleBench offers a public dataset and a "Try Yourself" option, plus the models via the Simple Bench app on LMcouncil.ai — useful if you want to see whether the trick questions fool you too.