What Is an LLM Judge and How Do You Use One to Score Model Outputs?

An LLM judge is a model that grades another model's output against criteria or a reference answer instead of a human doing it by hand. You use one when you need to compare models, prompts, or whole workflows on your own inputs and can't eyeball every result — for example, scoring 30 outputs across six models. The catch is that a single judge can be biased toward its own family or writing style, so the more reliable setup is a blind panel of judges plus a reference answer set, with scores read alongside cost and speed rather than on their own.

What an LLM judge actually does

A judge takes three things as input:

• The task input (the prompt, text, or listing being processed)
• The candidate output (what the model under test produced)
• A grading target — either explicit criteria ("is the language correctly detected?") or a gold/reference answer set

It returns a score, usually on a 0–1 scale or a small rubric. That score is what lets you rank models on your data instead of a public leaderboard that may not resemble your traffic.

This is different from a benchmark you download. A judge is the scoring mechanism; the benchmark is the input set you run through it.

Why blind judging and panels matter

A judge that knows which model produced an output can favor familiar phrasing, longer answers, or its own family's style. Blind judging hides the source so the score reflects the output, not the label.

A single judge still carries its own preferences, so Evaligo's published experiments use a blind three-judge panel — multiple judges score the same output and the results are combined. That reduces the chance one model's quirks decide your ranking.

You can see the effect in their own numbers. In a listing-localization experiment (63 listings, 7 languages, scored by a blind three-judge panel):

The top score and a near-tie 0.006 below it differ by roughly 20× in cost. A judge score alone would pick the winner; the judge score plus cost picks the model you'd actually ship.

Setting up judging criteria and reference answers

1. Define the input set. Use real tasks, not toy prompts. Evaligo's language-detection run used 129 texts; the writing-level run used 87 texts. Small but representative beats large and generic.
2. Write or collect a reference answer set where the task has a correct answer. The language-detection and writing-level experiments were "scored against a gold answer set" and "graded against a reference answer set" respectively — that's what makes 0.99 meaningful rather than arbitrary.
3. State the criteria explicitly for open-ended tasks (tone, accuracy, format compliance). Vague criteria produce noisy scores.
4. Run every variant on the same inputs. This is the fairness rule: same inputs, same judge panel, same scale. Evaligo's flow builder runs a flow on several models as variants so each one sees identical inputs.
5. Score blind, with more than one judge, then combine.

Practical pitfalls

• Self-preference. A judge tends to rate outputs that resemble its own. Blind panels dilute this; a single unblinded judge amplifies it.

• Price ≠ quality. In Evaligo's writing-level experiment (87 texts), the cheaper Flash model scored higher than the Pro model of the same family:

  Model	Score	Cost per text
  gemini-3.5-flash	0.977	$0.00648
  gemini-3.1-pro	0.948	$0.00774
  gemini-3-flash	0.937	$0.00175
  gemini-2.5-pro	0.845	$0.01151

  If you assume the pricier model wins, you overpay for worse output.

• Ceilings hide differences. In the language-detection run, every model scored 0.99 or better on the same 129 texts — the most expensive cost 32× the cheapest and did not score higher. When scores cluster at the top, the judge has stopped discriminating and cost becomes the deciding factor.

• Judging costs money too. Every judge call is a model call. A three-judge panel triples that. Budget for it before scaling the input set.

• Validate the judge. Before trusting scores, check that the judge agrees with a handful of human-labeled examples. If it doesn't, fix the criteria or the reference set first.

From scores to a decision

Judge scores are one column, not the answer. Read them next to cost and speed per output, then decide per input type:

• Scores tied, costs far apart → take the cheapest that clears your quality bar.
• Small score gap, large cost gap → the cheaper model is usually the right default; reserve the top scorer for inputs where the gap matters.
• Scores clustered near the ceiling → the task is easy for every model; optimize on cost and latency.

Evaligo's flow builder supports this directly: run a flow on several models, score each output with blind AI judges, compare quality, cost, and speed, then train a per-input model router on the results and deploy the flow as an API. The router is the payoff — instead of one model for everything, each input type gets the model that scored well enough at an acceptable cost.

Quick checklist

• Same inputs for every variant
• Reference answer set or explicit criteria
• Blind scoring, multiple judges
• Scores reported with cost and speed, not alone
• Judge validated against human labels before you trust it
• Decision made per input type, not globally