What Is LLM A/B Testing and How Do You Compare Models and Prompts Fairly?

LLM A/B testing means running the same set of inputs through two or more variants — different models, different prompts, or different multi-step flows — and scoring every output on the same task so you can compare quality, cost, and speed side by side. It's worth doing whenever you're about to commit to a model or prompt for a production workflow, because the cheapest option is often not the worst and the most expensive is often not the best. The catch: the comparison is only meaningful if the inputs are fixed, the task is identical across variants, and scoring is done blind or against a gold answer set rather than by eyeballing outputs.

Why price doesn't predict quality

Evaligo published three experiments on its own workflows, each running every model on the same inputs and recording score plus cost per output. The pattern across all three: quality clustered tightly while cost spread widely.

Language detection — 129 texts. Every model scored 0.99 or better. The most expensive cost 32× the cheapest and did not score higher.

Listing localization — 63 listings, 7 languages. Claude Sonnet 5 scored highest, but gpt-5.6-luna came within 0.006 of it for roughly a twentieth of the price per listing.

Writing level — 87 texts. The cheaper model graded better: gemini-3.5-flash scored 0.977 at $0.00648 per text, beating gemini-3.1-pro at 0.948 and $0.00774, and well above gemini-2.5-pro at 0.845 and $0.01151.

The takeaway isn't "always pick the cheap model." It's that you can't infer the ranking from the price tag — you have to measure it on your task.

Setting up a fair test

A comparison is only as good as its controls. Four things need to be fixed:

• The input set. Every variant sees the identical inputs. Thirty inputs is a reasonable starting floor; the language-detection experiment used 129, the localization one 63. Too few inputs and one lucky or unlucky case dominates the result.
• The task. All variants do the same job on the same inputs. Comparing model A on summarization against model B on translation tells you nothing.
• The variant list. Include open-source models alongside proprietary ones. Evaligo supports OpenAI, Google, and Anthropic models plus open-source options like DeepSeek, MiniMax, Kimi, and Qwen through a single OpenRouter key, so the open-source-vs-GPT comparison happens in the same run rather than as a separate afterthought.
• The scoring method. See below — this is where most homegrown tests go wrong.

In a visual builder like Evaligo's, you describe the flow in one sentence and it assembles the prompts, scraping, extraction, tools, and agents as connected nodes; you then drag nodes on the canvas or change the flow by instruction. Each model you want to compare becomes a variant of that same flow.

Scoring: blind judges or a gold answer set

Two approaches show up in the published experiments, and they answer slightly different questions.

Gold answer set. You have known-correct outputs, and each model's output is scored against them. The language-detection and writing-level experiments used this — hence scores like 1.000 and 0.977. Use it when correctness is objectively checkable.

Blind AI judge panel. The localization experiment used a blind three-judge panel, where judges score outputs without knowing which model produced them. Use it when quality is a matter of judgment (tone, fluency, localization naturalness) and no single gold answer exists.

Blindness matters in both cases. If a judge — human or model — can see which variant produced an output, brand familiarity and price bias leak into the score. The point of the panel is to make the model identity invisible at scoring time.

Reading the scoreboard

Once you have scores, costs, and latencies, read them together rather than in sequence:

1. Filter by a quality floor. Decide the minimum score your task tolerates, then discard everything below it. In the language-detection case, a 0.99 floor eliminates nothing — which is itself the finding.
2. Rank the survivors by cost per output. Among models that clear the floor, the cheapest is usually the right default.
3. Check latency. A model that's cheap and accurate but too slow for an interactive flow may still lose to a faster, slightly pricier option.
4. Look for near-ties. A 0.006 gap at a twentieth of the price (the localization result) is a strong signal to take the cheaper model; a large gap at a small price difference is a signal to pay up.

From results to routing and deployment

The results don't have to become a single global choice. Because quality varies by input type, you can train a per-input model router on the test results — sending easy inputs to the cheap model and harder ones to the stronger model — and then deploy the winning flow as an API. That way the A/B test isn't a one-time decision document; it becomes the routing logic itself.

Common pitfalls

• Too few inputs. A handful of examples produces a ranking that won't survive contact with real traffic.
• Non-blind judging. If the scorer knows which model it's grading, the scores drift toward expectations.
• Different tasks per variant. Any comparison where variants don't face identical inputs and identical jobs is measuring something other than model quality.
• Assuming price equals quality. All three published experiments contradict this — including one where the cheaper model graded better than the Pro model in the same family.
• Ignoring cost and speed. A quality-only scoreboard hides the 32× cost spread that a combined view makes obvious.

Evaligo offers a free plan with no credit card to run a first A/B test, and the full write-ups of the three experiments above are published on the site if you want to see the raw scoreboards before building your own.