evaligo.com

An LLM judge is a model that grades another model's output against criteria or a reference answer instead of a human doing it by hand. You use one when you need to compare models, prompts, or whole workflows on your own inputs and can't eyeball every result — for example, scoring 30 outputs across six models. The catch is that a single judge can be biased toward its own family or writing style, so the more reliable setup is a blind panel of judges plus a reference answer set, with scores read alongside cost and speed rather than on their own.

What an LLM judge actually does

A judge takes three things as input:

• The task input (the prompt, text, or listing being processed)
• The candidate output (what the model under test produced)
• A grading target — either explicit criteria ("is the language correctly detected?") or a gold/reference answer set

It returns a score, usually on a 0–1 scale or a small rubric. That score is what lets you rank models on your data instead of a public leaderboard that may not resemble your traffic.

This is different from a benchmark you download. A judge is the scoring mechanism; the benchmark is the input set you run through it.

Why blind judging and panels matter

A judge that knows which model produced an output can favor familiar phrasing, longer answers, or its own family's style. Blind judging hides the source so the score reflects the output, not the label.

A single judge still carries its own preferences, so Evaligo's published experiments use a blind three-judge panel — multiple judges score the same output and the results are combined. That reduces the chance one model's quirks decide your ranking.

You can see the effect in their own numbers. In a listing-localization experiment (63 listings, 7 languages, scored by a blind three-judge panel):

The top score and a near-tie 0.006 below it differ by roughly 20× in cost. A judge score alone would pick the winner; the judge score plus cost picks the model you'd actually ship.

Setting up judging criteria and reference answers

1. Define the input set. Use real tasks, not toy prompts. Evaligo's language-detection run used 129 texts; the writing-level run used 87 texts. Small but representative beats large and generic.
2. Write or collect a reference answer set where the task has a correct answer. The language-detection and writing-level experiments were "scored against a gold answer set" and "graded against a reference answer set" respectively — that's what makes 0.99 meaningful rather than arbitrary.
3. State the criteria explicitly for open-ended tasks (tone, accuracy, format compliance). Vague criteria produce noisy scores.
4. Run every variant on the same inputs. This is the fairness rule: same inputs, same judge panel, same scale. Evaligo's flow builder runs a flow on several models as variants so each one sees identical inputs.
5. Score blind, with more than one judge, then combine.

Practical pitfalls

• Self-preference. A judge tends to rate outputs that resemble its own. Blind panels dilute this; a single unblinded judge amplifies it.

• Price ≠ quality. In Evaligo's writing-level experiment (87 texts), the cheaper Flash model scored higher than the Pro model of the same family:

  Model	Score	Cost per text
  gemini-3.5-flash	0.977	$0.00648
  gemini-3.1-pro	0.948	$0.00774
  gemini-3-flash	0.937	$0.00175
  gemini-2.5-pro	0.845	$0.01151

  If you assume the pricier model wins, you overpay for worse output.

• Ceilings hide differences. In the language-detection run, every model scored 0.99 or better on the same 129 texts — the most expensive cost 32× the cheapest and did not score higher. When scores cluster at the top, the judge has stopped discriminating and cost becomes the deciding factor.

• Judging costs money too. Every judge call is a model call. A three-judge panel triples that. Budget for it before scaling the input set.

• Validate the judge. Before trusting scores, check that the judge agrees with a handful of human-labeled examples. If it doesn't, fix the criteria or the reference set first.

From scores to a decision

Judge scores are one column, not the answer. Read them next to cost and speed per output, then decide per input type:

• Scores tied, costs far apart → take the cheapest that clears your quality bar.
• Small score gap, large cost gap → the cheaper model is usually the right default; reserve the top scorer for inputs where the gap matters.
• Scores clustered near the ceiling → the task is easy for every model; optimize on cost and latency.

Evaligo's flow builder supports this directly: run a flow on several models, score each output with blind AI judges, compare quality, cost, and speed, then train a per-input model router on the results and deploy the flow as an API. The router is the payoff — instead of one model for everything, each input type gets the model that scored well enough at an acceptable cost.

Quick checklist

• Same inputs for every variant
• Reference answer set or explicit criteria
• Blind scoring, multiple judges
• Scores reported with cost and speed, not alone
• Judge validated against human labels before you trust it
• Decision made per input type, not globally

Run the same inputs through every model you're considering, score the outputs blind or against a reference answer set, and record quality, cost, and speed per output. Public leaderboards tell you how models perform on someone else's tasks; this method tells you which model wins on yours. It's worth doing whenever your workload is specific enough that a general ranking might not transfer — for example, language detection, listing localization, or grading writing level.

Why public benchmarks aren't enough

A model that tops a general leaderboard can lose on your task, and a cheaper model can tie or beat an expensive one. Evaligo's own published experiments show this pattern clearly:

The takeaway across all three: price did not predict quality. That's the reason to test on your own inputs rather than trust a ranking.

The core method: same inputs, multiple models, one scoreboard

The fair-comparison rule is simple — every model must see identical inputs, and every output must be scored the same way. Evaligo's flow builder is built around this: you build a flow of prompts, agents, and tools, then run it on several models as variants.

Step 1 — Build the flow

Describe the flow in one sentence and the builder assembles it: prompts, web scraping, data extraction, tools, and agents as connected nodes. You can drag nodes on the canvas to edit, or change the flow by instruction. The point is that the flow stays constant while the model changes — that's what makes the comparison valid.

Step 2 — Pick models as variants

Select the models you want to compare. This includes OpenAI, Google, and Anthropic models, plus open-source options such as DeepSeek, MiniMax, Kimi, and Qwen, reachable through a single OpenRouter key. Each model becomes a variant of the same flow.

Step 3 — Score every output blind

Use blind AI judge panels, or grade against a gold/reference answer set. Blind judging reduces the bias you'd get from knowing which model produced which output. In the localization experiment, a blind three-judge panel produced the scores.

Step 4 — Read the scoreboard

For each output you get quality, cost, and speed together — not just a score. This is what lets you catch the "same score, 32× the price" case, which a quality-only view would hide.

Step 5 — Route and deploy

Once you know which model wins for which input type, you can train a per-input model router on the results, then deploy the flow as an API. Routing matters when no single model wins everywhere — you send each input to the model that performs best on that kind of input.

What to record, and why each metric matters

A model that ties on score but costs a fraction as much is usually the right default; keep the expensive model only for input types where it measurably wins.

Common pitfalls

• Different inputs per model. If models don't see identical inputs, the comparison is meaningless.
• Unblinded scoring. Knowing which model produced an output biases the judgment.
• Quality-only evaluation. You'll miss cases where a cheap model ties an expensive one.
• Assuming price predicts quality. The writing-level experiment showed the opposite.
• Picking one model for everything. Per-input routing often beats a single global choice.

Getting started

Evaligo offers a free plan with no credit card, and you can run your first A/B test free. If you want to see the method in action before building your own, the published benchmarks (six models, the same 30 inputs, three blind judges) are available as a full write-up.

OpenAPI-generated API documentation is reference documentation that is produced automatically from an OpenAPI description file rather than written by hand. You write (or generate) a machine-readable specification of your API — endpoints, parameters, request bodies, responses, schemas, and auth — and a documentation tool reads that file and renders a browsable, often interactive reference site. The spec becomes the single source of truth; the docs become a build artifact.

This differs from manually written docs in one fundamental way: with hand-written docs, the prose is the source of truth and the API is described separately. With spec-driven docs, the API description is the source, and every page, table, and code sample is derived from it.

How the workflow actually runs

A typical spec-driven documentation pipeline has five stages:

1. Author or generate the spec. You either write an OpenAPI document by hand (YAML or JSON), or generate it from code annotations, framework metadata, or a design-first editor. Design-first means the spec is written before implementation; code-first means it is extracted from existing code.
2. Validate and lint. The spec is checked against the OpenAPI schema and against style rules — consistent naming, required descriptions, no undocumented 4xx responses, no orphaned schemas.
3. Bundle and transform. Multi-file specs are combined, $ref pointers are resolved, and the document is optionally split into per-tag or per-version outputs.
4. Render. A documentation tool converts the spec into HTML: an endpoint list, a sidebar of operations, parameter tables, response schemas, and a "try it" console.
5. Publish and version. The rendered site is deployed, and each API version gets its own snapshot so consumers can read docs matching the version they call.

Steps 2 through 5 are usually automated in CI. If the spec fails validation, the docs build fails — which is the point.

Spec-driven vs. hand-written documentation

The practical conclusion most teams reach: generate the reference, write the guides. Reference material is repetitive and mechanical, which is exactly what generation is good at. Conceptual explanations, migration notes, and tutorials carry judgment that a spec cannot express.

What you get out of the box

Generated reference pages commonly include:

• An operation list grouped by tag or path, with HTTP method and path.
• Parameter tables showing name, location (path, query, header, cookie), type, required flag, and description.
• Request and response schemas rendered as expandable trees, including nested objects and arrays.
• Authentication details pulled from the securitySchemes section.
• Interactive request consoles that let a reader send a real call from the browser.
• Generated code samples in several languages, derived from the same schemas.
• Multiple output formats, such as a static site, a single HTML file, or a mock server.

Because all of these come from one document, changing a field name in the spec updates the parameter table, the schema tree, and every code sample at once.

Where spec-driven documentation breaks down

Generation is not free. The trade-offs are real:

Spec quality becomes documentation quality. A field with no description produces a table row with an empty cell. A vague summary produces a vague heading. Tools can enforce presence of descriptions via linting, but they cannot enforce that the description is useful.

Customization has limits. If you need a page that does not map to an OpenAPI concept — a conceptual overview, a pricing explanation, a comparison of two endpoints — you write it outside the generator and link to it.

Not everything is expressible. Webhooks, streaming responses, long-polling behavior, and complex multi-step flows are awkward or impossible to describe fully in OpenAPI. Those need prose.

The spec can go stale. If the spec is maintained separately from the implementation, it drifts just like hand-written docs. The mitigation is to generate the spec from code, or to test the implementation against the spec in CI.

Interactive consoles need care. A "try it" button that hits a production API with real credentials is a security and rate-limit problem. Point it at a sandbox, or disable it.

Deciding whether to adopt it

Adopt spec-driven reference documentation if most of these are true:

• Your API has more than a handful of endpoints, or changes frequently.
• You ship client SDKs or code samples in more than one language.
• Multiple teams consume the API and need a consistent, always-current reference.
• You already have, or are willing to maintain, an OpenAPI description.

Stay with hand-written docs, or a hybrid, if:

• Your API is small and stable, and the reference fits on one page.
• Your documentation is mostly conceptual and contains little endpoint-level detail.
• You cannot commit to keeping the spec in sync with the implementation.

A reasonable middle path: generate the reference from the spec, and hand-write the getting-started guide, authentication walkthrough, and error-handling page. Link the two directions so readers can move from concept to endpoint and back.

A minimal starting checklist

1. Produce one valid OpenAPI document for a single API version.
2. Add a linter with rules for descriptions, operation IDs, and error responses.
3. Wire the docs build into CI so a failing spec fails the build.
4. Render the reference and review it as a reader, not as the author.
5. Write the two or three conceptual pages the generator cannot produce.
6. Version the published docs alongside the API version.

The core idea is simple: describe the API once, in a format both machines and humans can read, and let the reference documentation fall out of that description. Everything else — tooling, hosting, interactivity — is a detail on top of that decision.

An AI workflow builder is a visual canvas where you connect prompts, tools, agents, and data steps into a single flow, then run that flow across multiple AI models to compare quality, cost, and speed before deploying it as an API. You'd use one when a task needs more than a single prompt call—for example, scraping a page, extracting fields, and grading the result—and when you want evidence, not guesswork, about which model to run in production.

The core loop: build, compare, score, deploy

Most prompt tools stop at "write a prompt and pick a model." A workflow builder adds three things on top of that:

1. A flow, not a prompt. Prompts, web scraping, data extraction, tools, and agents become connected nodes on a canvas. You can drag nodes to edit them or describe the flow in one sentence and have it built for you.
2. The same inputs across many models. Each model becomes a variant of the flow, so every variant sees identical inputs. That's what makes a comparison fair.
3. Scoring and deployment. Outputs get scored, you read a scoreboard, and any flow can ship as an API.

The loop is: build a flow → run it on several models with the same inputs → score every output → read the results → deploy the winner.

Running the same inputs across models

This is the step that separates a builder from a chat interface. You pick the models you want to compare—OpenAI, Google, and Anthropic, plus open-source options like DeepSeek, MiniMax, Kimi, and Qwen through one OpenRouter key—and each one runs as a variant on the same inputs.

Because inputs are held constant, differences in the output come from the model, not from the test. That's the condition you need before any cost or quality claim means anything.

Blind AI judges and why "blind" matters

Scoring is done by blind AI judge panels: the judges don't know which model produced which output. This removes the most common source of bias in model comparison—knowing the brand behind an answer and grading it accordingly.

Evaligo's own published experiments show why this matters. In a language-detection test on 129 texts, six models all scored 0.99 or better, but the most expensive cost 32× the cheapest without scoring higher:

Two other experiments point the same way. In listing localization (63 listings, 7 languages, blind three-judge panel), Claude Sonnet 5 scored highest at 0.914, but gpt-5.6-luna came within 0.006 for a twentieth of the price per listing. In a writing-level test on 87 texts, the cheaper Flash model outscored the Pro model of the same family and cost less per text.

The takeaway across all three: price did not predict quality. You can't pick a model from a price list—you have to measure it on your own inputs.

From scoreboard to production: per-input model routing

A scoreboard tells you which model won on average. Per-input model routing goes further: the builder trains a router on the comparison results so each individual input is sent to the model that performs best for that kind of input. Instead of one model for everything, you route by input.

This is the practical payoff of running the same inputs across many models—the results become a routing policy, not just a report.

Deploying a flow as an API

Once a flow is built and tested, you can deploy it as an API. That means the flow you validated in the builder is the same thing you call from your own application, rather than a re-implementation that drifts from what you tested. The comparison work stays attached to the thing you ship.

A concrete example

Say you need to localize product listings into seven languages. Build a flow: extract the listing fields, generate the localized copy, and score the output. Run it on four or five models with the same 60 listings. A blind judge panel scores each output, and you see quality and cost per listing side by side. If a mid-priced model lands within a few thousandths of the top scorer at a fraction of the cost, route to it—or let per-input routing decide case by case—then deploy the flow as an API.

What to check before choosing a builder

• Does it hold inputs constant across models? Without this, comparisons are noise.
• Are judges blind? Brand-aware grading inflates known models.
• Does it report cost and speed per output, not just quality? All three decide production fit.
• Can results become routing, or only a report? Per-input routing is what turns a test into a decision.
• Can you deploy the tested flow directly as an API? Otherwise you re-build and lose the validation.
• Does it cover open-source models? The cheapest winner is often not a frontier model.

Evaligo offers a free plan with no credit card, so the build-compare-score loop is something you can run before committing.

An AI model benchmark is a repeatable test that scores model outputs against a known standard. A public benchmark scores every model on the same fixed dataset that someone else chose; a benchmark on your own tasks scores the models you're considering on inputs from your actual workflow, judged against answers you trust. The second kind is what tells you which model to ship. You can run one whenever you can assemble a set of representative inputs and a way to grade the outputs — either reference answers or blind judges.

Public leaderboards vs. your own benchmark

Public leaderboards answer "how do these models do on a shared, generic task set?" That's useful for narrowing a shortlist, but it rarely matches your inputs, your prompt style, or your quality bar. A model that tops a leaderboard can lose on your data, and a cheaper model can tie it.

The difference comes down to four things:

The practical move is to use public results to build a shortlist, then run your own benchmark to pick the winner.

What a fair benchmark requires

Four conditions separate a real comparison from a vibe check:

1. Same inputs for every model. If model A sees 30 examples and model B sees 12 different ones, you've measured the inputs, not the models.
2. Same prompt and flow. Changing the prompt between models confounds the result — you won't know whether the score moved because of the model or the instructions.
3. Blind scoring. The judge shouldn't know which model produced which output. This is the main defense against a judge favoring a familiar name or a longer answer.
4. Quality measured next to cost and speed. A score alone doesn't tell you whether the extra quality is worth the extra spend.

Evaligo's own published experiments follow this shape: every model runs the same inputs, and each output gets a score and a cost figure. In one language-detection test, six models ran the same 129 texts and every one scored 0.99 or better — but the most expensive cost 32× the cheapest without scoring higher. That result is only visible because cost was recorded per output next to the score.

How to run one, step by step

1. Collect representative inputs

Pull a sample of real inputs from the task you care about — support tickets, product listings, user messages, documents. Aim for enough variety that a model can't win by luck on a narrow slice. The published Evaligo experiments used sets like 129 texts, 63 listings across 7 languages, and 87 texts; small sets are fine to start, but a handful of examples won't separate close models.

2. Define your grading standard

Pick one of two approaches:

• Reference answer set — you have known-correct outputs to score against. Good for tasks with a defensible right answer (language detection, structured extraction).
• Blind judge panel — one or more AI judges score each output without seeing which model made it. Good for open-ended quality (writing level, localization tone).

A panel of several judges reduces the chance that one judge's quirks drive the result. Evaligo's localization experiment used a blind three-judge panel.

3. Build the flow and add model variants

In a workflow builder, you describe the flow — prompts, scraping, extraction, tools, agents as connected nodes — and then attach each model you want to test as a variant. Evaligo supports OpenAI, Google, and Anthropic models plus open-source options such as DeepSeek, MiniMax, Kimi, and Qwen through a single OpenRouter key. The point of variants is that the flow stays identical while only the model changes.

4. Run and read the scoreboard

Each run produces a score per output plus cost and speed. Read it in this order:

• Filter by quality floor. Drop any model below the score you need.
• Among survivors, compare cost per output. This is where the surprises live.
• Check speed if latency matters to your users.

In the localization test, Claude Sonnet 5 scored highest at 0.914, but gpt-5.6-luna came within 0.006 at a twentieth of the cost per listing. In the writing-level test, a Flash model outscored the Pro model of the same family and cost less per text — price did not predict quality.

5. Decide: one model, or routing

If one model wins on both quality and cost, deploy it. If different models win on different input types, you can train a per-input router on the benchmark results so each input goes to the model that handles it best. Either way, the flow can be deployed as an API once you've chosen.

Common pitfalls

• Too few inputs. A 5-example test can't distinguish models that differ by a few points. The published experiments used 63–129 inputs per task.
• A single judge. One judge's bias becomes your result. Use a panel when grading is subjective.
• Different prompts per model. You'll attribute a prompt difference to the model. Keep the flow fixed and vary only the model.
• Ignoring cost. A 32× price gap for an identical score is a real finding, and it only shows up if you record cost per output.
• Trusting price as a proxy for quality. The writing-level experiment found the cheaper model graded better than the pricier one in the same family.
• Benchmarking once and never again. Models and prompts change; re-run when you swap either.

What you get out of it

A benchmark on your own tasks replaces "this model seems good" with a scoreboard: which model clears your quality bar, what each output costs, and how fast it runs. From there you either pick a single model or route inputs to the best one per case, and deploy the flow as an API. Evaligo offers a free plan with no credit card to run a first A/B test, and its published benchmarks are available to read as worked examples of the method.