简体中文
EnglishIs Vals AI's Evaluation Independent and Unbiased?
Updated
Vals AI presents itself as an independent evaluator: it states that it runs all of its own evaluations and creates many of its benchmarks in-house, rather than republishing vendor-run results. That claim is the basis for treating its leaderboard as a third-party signal. The practical caveat is that "independent" describes who runs the tests, not a guarantee of neutrality — you still need to check what each benchmark measures before using a score to make a decision.
What "independent" means on vals.ai
The site's own framing is explicit: "Independent Evaluation, Unbiased Benchmarks." Two operational details back that up in the page evidence:
- Self-run evaluations. Vals AI says it runs all of its own evaluations, so scores are not simply supplied by model vendors.
- In-house benchmarks. It creates many of its benchmarks itself, which matters because a benchmark written by the evaluator can be designed around real tasks rather than reused public test sets.
The scope is deliberately applied rather than general: models are benchmarked on "economically valuable tasks such as finance, software, and frontier risk like cybersecurity, recursive self improvement and mental health." That tells you the intended use — comparing models for work-like tasks, not for casual chat quality.
Why it argues third-party measurement is needed
The page makes a specific methodological argument about search evaluation, and it generalizes to the rest of the suite:
- Data contamination. If benchmark questions and answers circulate publicly, models can be trained on them, and scores stop reflecting capability.
- Unrealistic tasks. Tests that don't resemble actual work can rank tools in an order that doesn't transfer to real use.
- Answers leaking onto the web. Once expected outputs are findable, a model can retrieve rather than solve.
Vals AI's stated response is that search providers have built their own benchmarks to work around these problems, and that customers therefore can't tell which tool is best for their work without a third-party benchmark that measures the right thing. The same logic is why it builds benchmarks in-house instead of relying only on inherited test sets.
What the leaderboard actually reports
The Industry Leaderboard shows one best model per lab, with a headline performance score and additional dimensions:
| Dimension | What it tells you |
|---|---|
| Performance Score | Headline accuracy on the benchmark suite |
| Cost | What running the model costs |
| Latency | Response speed |
| Token Usage | How much output the model consumes |
As of the Oct 02, 2026 snapshot in the page evidence, the top of the list reads: Gemini 4 Argon 68.90%, Claude Sonnet 5.5 67.04%, GPT-6 Astra 63.13%, Muse Spark 1.3 Max 58.16%, MiMo V2.6 Pro 55.20%, Grok 4.7 54.95%, down to Mercury 2.5 at 8.86%. The spread is wide enough that the ranking is informative at the top and mostly a floor signal at the bottom.
Because cost, latency, and token usage sit alongside the score, the leaderboard is built for trade-off decisions rather than a single "best model" answer. A model a few points behind on performance may still be the right pick if its cost or latency profile fits your workload.
How to judge the independence claim for yourself
Independence is a property you verify per benchmark, not a label you accept once. A workable check:
- Identify which benchmark produced the score you care about. The suite spans finance, software, cybersecurity, and frontier-risk areas; a strong aggregate doesn't mean every sub-benchmark is equally mature.
- Look for the inputs and outputs. Vals AI's report on Claude Opus 5.5 agents finding magnetic semiconductor candidates states that the inputs, raw outputs, and analysis code behind the post's numbers are in a ledger linked from the post. Reproducibility material like that is the strongest available evidence that a result wasn't hand-picked.
- Check whether the task resembles yours. A benchmark can be independent and still measure something you don't do.
- Read the score with the other columns. Performance without cost, latency, and token usage is an incomplete basis for a deployment choice.
If a benchmark publishes its task design and raw outputs, you can audit the claim. If it only publishes a number, independence is asserted rather than demonstrated.
Where the limits are
- "Unbiased" is a design goal, not an audited certification. The page states the intent; it doesn't describe an external audit of the benchmark design.
- In-house benchmarks trade one risk for another. Avoiding contaminated public test sets is a real gain, but benchmarks written by the evaluator reflect that evaluator's judgment about what counts as a good task.
- Coverage is uneven by design. Finance, software, cybersecurity, and frontier-risk areas are the stated focus; other domains may be thinner or absent.
- Scores are snapshots. The leaderboard is dated (Oct 02, 2026 in the evidence shown), and new model evaluations appear frequently, so a ranking can shift between your reading and your decision.
The reasonable conclusion: Vals AI's self-run, in-house approach is a genuine third-party signal rather than vendor marketing, and its reproducibility practices in at least some reports let you check the work. Treat the independence claim as verified per benchmark — by finding the task design and raw outputs — rather than as a blanket property of every number on the site.