vals.ai

Vals AI presents itself as an independent evaluator: it states that it runs all of its own evaluations and creates many of its benchmarks in-house, rather than republishing vendor-run results. That claim is the basis for treating its leaderboard as a third-party signal. The practical caveat is that "independent" describes who runs the tests, not a guarantee of neutrality — you still need to check what each benchmark measures before using a score to make a decision.

What "independent" means on vals.ai

The site's own framing is explicit: "Independent Evaluation, Unbiased Benchmarks." Two operational details back that up in the page evidence:

• Self-run evaluations. Vals AI says it runs all of its own evaluations, so scores are not simply supplied by model vendors.
• In-house benchmarks. It creates many of its benchmarks itself, which matters because a benchmark written by the evaluator can be designed around real tasks rather than reused public test sets.

The scope is deliberately applied rather than general: models are benchmarked on "economically valuable tasks such as finance, software, and frontier risk like cybersecurity, recursive self improvement and mental health." That tells you the intended use — comparing models for work-like tasks, not for casual chat quality.

Why it argues third-party measurement is needed

The page makes a specific methodological argument about search evaluation, and it generalizes to the rest of the suite:

• Data contamination. If benchmark questions and answers circulate publicly, models can be trained on them, and scores stop reflecting capability.
• Unrealistic tasks. Tests that don't resemble actual work can rank tools in an order that doesn't transfer to real use.
• Answers leaking onto the web. Once expected outputs are findable, a model can retrieve rather than solve.

Vals AI's stated response is that search providers have built their own benchmarks to work around these problems, and that customers therefore can't tell which tool is best for their work without a third-party benchmark that measures the right thing. The same logic is why it builds benchmarks in-house instead of relying only on inherited test sets.

What the leaderboard actually reports

The Industry Leaderboard shows one best model per lab, with a headline performance score and additional dimensions:

As of the Oct 02, 2026 snapshot in the page evidence, the top of the list reads: Gemini 4 Argon 68.90%, Claude Sonnet 5.5 67.04%, GPT-6 Astra 63.13%, Muse Spark 1.3 Max 58.16%, MiMo V2.6 Pro 55.20%, Grok 4.7 54.95%, down to Mercury 2.5 at 8.86%. The spread is wide enough that the ranking is informative at the top and mostly a floor signal at the bottom.

Because cost, latency, and token usage sit alongside the score, the leaderboard is built for trade-off decisions rather than a single "best model" answer. A model a few points behind on performance may still be the right pick if its cost or latency profile fits your workload.

How to judge the independence claim for yourself

Independence is a property you verify per benchmark, not a label you accept once. A workable check:

1. Identify which benchmark produced the score you care about. The suite spans finance, software, cybersecurity, and frontier-risk areas; a strong aggregate doesn't mean every sub-benchmark is equally mature.
2. Look for the inputs and outputs. Vals AI's report on Claude Opus 5.5 agents finding magnetic semiconductor candidates states that the inputs, raw outputs, and analysis code behind the post's numbers are in a ledger linked from the post. Reproducibility material like that is the strongest available evidence that a result wasn't hand-picked.
3. Check whether the task resembles yours. A benchmark can be independent and still measure something you don't do.
4. Read the score with the other columns. Performance without cost, latency, and token usage is an incomplete basis for a deployment choice.

If a benchmark publishes its task design and raw outputs, you can audit the claim. If it only publishes a number, independence is asserted rather than demonstrated.

Where the limits are

• "Unbiased" is a design goal, not an audited certification. The page states the intent; it doesn't describe an external audit of the benchmark design.
• In-house benchmarks trade one risk for another. Avoiding contaminated public test sets is a real gain, but benchmarks written by the evaluator reflect that evaluator's judgment about what counts as a good task.
• Coverage is uneven by design. Finance, software, cybersecurity, and frontier-risk areas are the stated focus; other domains may be thinner or absent.
• Scores are snapshots. The leaderboard is dated (Oct 02, 2026 in the evidence shown), and new model evaluations appear frequently, so a ranking can shift between your reading and your decision.

The reasonable conclusion: Vals AI's self-run, in-house approach is a genuine third-party signal rather than vendor marketing, and its reproducibility practices in at least some reports let you check the work. Treat the independence claim as verified per benchmark — by finding the task design and raw outputs — rather than as a blanket property of every number on the site.

Vals AI is an independent evaluation platform that benchmarks leading AI models on economically valuable, real-world tasks rather than synthetic puzzles. It runs its own evaluations and builds many of its benchmarks in-house, covering domains such as finance, software, cybersecurity, healthcare, and frontier risks like recursive self-improvement and mental health. If you need a third-party read on how models perform on practical work—and you care about methodology independence—it's worth consulting. If you need vendor-specific pricing, deployment guidance, or guarantees about any single model's fitness for your workflow, it is not that kind of resource.

What Vals AI actually measures

The platform's stated focus is testing AI on tasks with economic value. Its benchmark suite spans:

• Finance — task-based evaluation of financial work
• Software — engineering and coding tasks
• Cybersecurity — including frontier-risk evaluation
• Healthcare — domain-specific task performance
• Frontier risk — recursive self-improvement and mental health

Two claims matter for how much weight to give the results: Vals AI runs all of its own evaluations, and it creates many of its benchmarks in-house. That distinguishes it from aggregators that republish vendor-reported numbers.

How results are presented

Each model entry is scored across four dimensions:

Presenting cost, latency, and token usage alongside accuracy is the practically useful part. A model that tops the scoreboard but costs several times more per task, or responds too slowly for interactive use, may not be the right pick—and the leaderboard lets you see that trade-off rather than a single accuracy number.

The leaderboard, as published

The most recent snapshot on the site (dated Oct 02, 2026) shows the best model from each lab:

Two things to note before you act on this. First, the spread is wide—from 68.90% down to 8.86%—so the benchmark is discriminating between models rather than clustering them. Second, a single aggregate score hides per-domain differences; a model ranked mid-table overall may still lead in the specific domain you care about. Check the full results rather than the summary ranking.

Reports and evaluations

Beyond the leaderboard, Vals AI publishes benchmark releases and model evaluations. Recent examples from the site include:

• Claude Opus 5.5 agent task (Oct 04, 2026) — agents were asked to find a room-temperature magnet with zero net magnetism whose electrons are still sorted by spin (a Luttinger-compensated magnet). The agents proposed YBaMnFeO5, which looked ideal on paper but their follow-up calculations suggested it may be hard to make; they then identified KV[Cr(CN)6], a magnet first made in 1999 whose sample stayed magnetic up to 376 K, as fitting the description in their calculations. The inputs, raw outputs, and analysis code behind the numbers are in a ledger linked from the post.
• Vals Web Search Index (Oct 02, 2026) — an evaluation of search for real work. The stated motivation: web search hasn't been measured well, and providers have built their own benchmarks to work around data contamination, unrealistic tasks, and answers leaking onto the web. When a benchmark measures the wrong thing, customers can't tell which search tool is best for their work.
• "Has the Bitter Lesson Come for AI Detectors?" (Sep 30, 2026)
• Gemini 4 Argon (Sep 30, 2026) and Claude Sonnet 5.5 (Sep 28, 2026) — evaluated across the benchmark suite.

The agent-magnet report is a useful example of the platform's approach: it publishes the inputs, raw outputs, and analysis code, so you can inspect the reasoning rather than trust a headline number.

When to use Vals AI—and when not to

Use it when you want to:

• Compare models on task performance rather than marketing claims
• Weigh accuracy against cost, latency, and token usage in one view
• See whether a model's strength holds in a specific domain (finance, software, cybersecurity, healthcare)
• Check a third-party evaluation before committing to a model for production work

Look elsewhere when you need:

• Pricing or plan details for the platform itself—the site's pricing signals aren't published, so don't assume free or paid access
• Implementation guidance, API documentation, or integration steps for any model
• A verdict on whether a model meets your compliance, security, or latency requirements specifically

How to read the results responsibly

Benchmarks are a starting point, not a decision. Three practical cautions:

1. Check the domain, not just the rank. The overall score aggregates across finance, software, cybersecurity, healthcare, and frontier risk. Your use case lives in one of those, so find the relevant breakdown.
2. Treat cost and latency as first-class. A 1.9-point accuracy gap between the top two models may be irrelevant if one is materially cheaper or faster for your workload.
3. Prefer evaluations with published artifacts. The reports that link their inputs, raw outputs, and analysis code are the ones you can actually audit. Where that transparency is missing, weight the number accordingly.

Vals AI's value is that it produces independent, in-house evaluations with visible methodology and multi-dimensional scoring. That makes it a reasonable input to a model-selection decision—provided you read past the headline ranking and into the domain-level detail.

Vals AI ranks models by a single headline number — a performance score expressed as a percentage — computed from its own benchmark suite on economically valuable tasks such as finance, software, and frontier risk areas like cybersecurity, recursive self-improvement, and mental health. The leaderboard is a comparison tool, not a certification: it tells you how models performed on Vals AI's tasks as of a given report date, and it shows only the best model from each lab by default, with a link to view full results.

What the ranking is actually measuring

Vals AI states that it runs all of its own evaluations and creates many of its benchmarks in-house. That matters for interpretation in two ways:

• The score is benchmark-specific. A model's percentage reflects performance on Vals AI's task set, not a universal capability measure. A model that leads on this leaderboard is not automatically the best choice for your particular workload.
• The suite spans several domains. The stated coverage includes finance, software, cybersecurity, recursive self-improvement, and mental health. Because the headline score aggregates across these areas, a model that is strong in one domain and weak in another can land mid-table.

The metrics shown alongside the score

The leaderboard presents more than a single ranking. Alongside performance score, it reports:

These four dimensions are the reason the leaderboard is useful for decisions rather than just curiosity. Two models with close scores can differ substantially on cost, latency, or token consumption — and for a production use case, that gap may matter more than a point or two of accuracy.

How to read the current standings

The most recent report shown on the site is dated Oct 02, 2026, with these top scores:

Note the spread: the top three cluster within about six points of each other, while the bottom of the table falls off sharply. The default view shows the best model from each lab, so a lab's second-best model will not appear unless you open the full results.

How to use it to make a decision

1. Start with the score band, not the exact rank. If two models are within a few points, treat them as roughly comparable on accuracy and decide on cost, latency, or token usage instead.
2. Check the report date. Scores are tied to a dated evaluation. A newer model release may not yet be evaluated, and older entries may not reflect current versions.
3. Match the domain to your task. If your work is in finance or software, look for the relevant benchmark report rather than relying only on the aggregate leaderboard.
4. Open the full results. The default leaderboard hides all but each lab's best model, so the full view is where you find alternatives and per-benchmark detail.

Where the leaderboard is not enough

The leaderboard answers "how did these models score on Vals AI's tasks on this date." It does not answer "which model is best for my specific pipeline." For that, you need either a domain-specific report from the benchmark suite or your own evaluation on representative inputs. The leaderboard is best used as a shortlist generator: pick the top few models within a reasonable score band, then test them against your actual workload.

Vals AI's latest reports cover two main things: benchmark releases and individual model evaluations. Topics span finance, software, cybersecurity, frontier risk (including recursive self-improvement and mental health), web search, and AI detectors. Models evaluated in the recent cycle include Gemini 4 Argon, Claude Sonnet 5.5, GPT-6 Astra, Claude Opus 5.5, and a long list of others ranked on the site's leaderboard.

Topics covered in recent reports

The site describes its benchmark scope as "economically valuable tasks such as finance, software, and frontier risk like cybersecurity, recursive self improvement and mental health." Recent report entries add two more specific areas:

• AI-driven scientific discovery — a report on Claude Opus 5.5 agents searching for room-temperature magnetic semiconductor candidates.
• Web search evaluation — the "Vals Web Search Index," framed around evaluating search for real work rather than synthetic tasks.
• AI detectors — a piece asking whether the "bitter lesson" has arrived for AI detector tools.

The web search report also states a rationale worth noting: search providers have built their own benchmarks to work around data contamination, unrealistic tasks, and answers leaking onto the web, and Vals AI argues customers need a third-party benchmark to tell which search tool is best for their work.

Models evaluated

The leaderboard snapshot lists the best model from each lab, with performance scores:

Individual evaluation reports exist for Gemini 4 Argon and Claude Sonnet 5.5, both described as "evaluated across our benchmark suite."

What each report includes

The magnetic semiconductor report is the most detailed example of what a report contains. It describes the task given to Claude Opus 5.5 agents (finding a room-temperature magnet with zero net magnetism whose electrons are still sorted by spin — a Luttinger-compensated magnet), the candidate the agents designed (YBaMnFeO5), the follow-up finding that it may be hard to make, and the alternative they identified (KV[Cr(CN)6], a magnet first made in 1999 whose sample stayed magnetic up to 376 K). The post links to a ledger containing the inputs, raw outputs, and analysis code behind its numbers.

That pattern — task description, model output, and a link to underlying data — is the clearest signal of what to expect from other reports on the site.

How to use this to judge coverage

If you want to know whether Vals AI has evaluated a specific model, check the leaderboard first; it shows one representative model per lab rather than every variant. If you want to know whether a specific capability is tested, look at the report titles — the site names the domain (web search, AI detectors, scientific discovery) rather than burying it. And if you need to verify a claim, the magnetic semiconductor report suggests at least some reports publish their inputs and analysis code, so look for a linked ledger before assuming a number is unauditable.