respan.ai Paid content

Prompt optimization is the practice of improving a prompt through repeated measurement rather than intuition. Instead of rewriting a prompt and hoping it works better, you define what "better" means, run the current prompt and its variants against a fixed set of test cases, and keep the version that scores highest. This matters most when a prompt is already in production, when multiple people edit it, or when small wording changes have caused silent regressions. It is less necessary for one-off prompts you will never reuse.

Prompt engineering vs. prompt optimization

These terms get used interchangeably, but they describe different activities.

Prompt engineering produces a candidate. Prompt optimization decides whether that candidate is actually better.

The optimization loop

The loop has four stages, and you repeat it until improvements stop mattering.

1. Define success criteria

Before touching the prompt, write down what a correct output looks like. Be specific enough that two people would agree on whether a given output passes. Useful criteria include:

• Task accuracy — did the model produce the right answer, classification, or extraction?
• Format compliance — did it follow the required structure (JSON, a fixed number of bullets, a specific field set)?
• Latency — how long does a run take end to end?
• Cost — how many tokens does the prompt and completion consume?

Pick the one or two criteria that actually matter for your use case. A support-triage prompt cares about classification accuracy; a summarization prompt may care more about format and length.

2. Build a small eval set

You do not need thousands of examples to start. Twenty to fifty representative inputs, each with an expected output or a pass/fail rule, will expose most prompt problems. Include:

• Common cases the prompt handles today
• Edge cases you have seen fail
• At least a few adversarial or ambiguous inputs

Keep this set fixed while you compare variants. If you change the test cases between runs, you cannot tell whether the prompt improved or the test got easier.

3. Compare variants

Change one thing at a time so you know what caused the difference. Run each variant against the same eval set and record the scores. A variant only wins if it improves your primary criterion without unacceptable regression on the others — a prompt that gains 5% accuracy but doubles latency may not be worth shipping.

4. Promote the best version

Once a variant wins, make it the production prompt and record what changed and why. This version history is what lets you roll back when a later edit turns out to be worse.

Techniques that usually move the score

These are the changes most likely to show up as measurable gains:

• Clarify instructions. Replace vague verbs with explicit ones. "Summarize this" becomes "Summarize this in exactly three sentences, each under 20 words."
• Add few-shot examples. Two or three well-chosen input/output pairs often fix format and tone problems faster than more instruction text.
• Specify an output schema. If you need structured output, state the exact fields and types rather than describing the shape in prose.
• Decompose the task. Split a prompt that asks for extraction, reasoning, and formatting into separate steps. Each step is easier to evaluate and debug.
• Constrain the scope. Tell the model what to do when the input does not fit — for example, return a specific error value instead of guessing.

Each of these is a hypothesis. Test it against the eval set rather than assuming it helps.

Where observability feeds back in

Evals tell you how a prompt performs on cases you chose. Observability tells you what happens on cases you did not choose. Run traces from production show the actual inputs, the prompt version used, the model's output, and how long each step took. When a prompt starts failing in production, traces show you the input that broke it.

That failing input becomes a new eval case. This is the feedback loop: production traces surface real failures, those failures join the eval set, and the next optimization round is measured against them. Without this loop, your eval set drifts away from reality and your scores stop predicting production behavior.

Platforms that combine routing, observability, and automated evals — Respan describes itself as an AI router with built-in observability and automated evals — are built around exactly this cycle, so that traces and eval results live in the same place as the prompt versions they describe.

Common pitfalls

• Overfitting to a handful of examples. If you tune the prompt until it passes your five test cases, it will fail on the sixth. Keep some cases out of the tuning set and use them only for final checks.
• Silent regressions. A wording change that fixes one case often breaks another. Always re-run the full eval set after any edit, not just the case you were fixing.
• No baseline. If you never scored the original prompt, you cannot prove the new one is better. Score first, then change.
• Optimizing the wrong metric. A prompt that maximizes accuracy but triples cost per call may be a net loss. Decide which trade-offs are acceptable before you start.
• Changing the eval set mid-comparison. This invalidates every comparison you made before the change.

A practical starting sequence

1. Pick one prompt that is already in use and matters.
2. Write down its success criteria and collect 20–50 test inputs with expected outputs.
3. Score the current prompt. This is your baseline.
4. Make one change, re-score, and keep the change only if the primary metric improves without breaking the others.
5. Repeat until gains flatten.
6. Ship the best version, record the change, and add production failures to the eval set as they appear.

The core discipline is simple: never accept a prompt change you cannot measure. Everything else — few-shot examples, schemas, decomposition — is a tool for generating candidates worth measuring.

An LLM engineering platform is the tooling layer that sits around your LLM or agent application and handles four jobs: routing requests to models, tracing what every run actually did, evaluating output quality automatically, and feeding those results back into prompt or model changes. You need one when your app talks to more than one model, runs agents in production, or has started shipping quality regressions you can't reproduce. If you're still on a single model with a handful of prompts and no users depending on it, ad-hoc scripts are usually enough.

The four capabilities that define the category

These are the pieces that show up together in a real platform. A product with only one of them is a point tool, not a platform.

1. Unified model gateway and routing

A gateway gives you one interface to reach every model provider, so switching or adding a model doesn't mean rewriting call sites. Respan describes itself as "the AI router," with the stated goal to "reach every model, and stay up" — meaning routing is also a resilience mechanism: when a provider degrades, traffic can move rather than fail.

What to verify: which providers are covered, whether routing rules can be based on cost, latency, or task type, and what happens to in-flight requests during a failover.

2. Observability and tracing of every run

Observability means you can see exactly what an agent did — which tools it called, in what order, with what inputs and outputs. Respan frames this as "see exactly what your agents did" and "know the moment things change." For agent applications this matters more than for single-prompt apps, because a bad answer is often a multi-step trajectory problem, not a single bad completion.

What to verify: whether traces capture full agent steps (not just the final LLM call), how long trace data is retained, and whether you can search traces by user, session, or failure type.

3. Automated evals

Evals are how you turn "this output looks worse" into a measurable signal. Automated evals run scoring against your outputs on a schedule or on every deploy, so regressions surface without a human reading samples. Respan pairs this with routing under the promise to "ship the version that scores best" — the eval result is meant to drive which prompt or model version goes live.

What to verify: whether you can define custom eval criteria or only use built-in scorers, whether evals run on live traffic or only on offline datasets, and how eval results connect back to a specific prompt or model version.

4. Prompt optimization

Once you can measure quality, you can improve it systematically instead of by guesswork. Prompt optimization closes the loop: eval scores identify weak prompts, changes get tested, and the winning version is promoted. This is the step most teams skip, which is why they plateau.

How the pieces fit into one workflow

The value is in the loop, not the individual features:

1. Route — a request enters through the gateway and goes to a chosen model or provider.
2. Trace — the full run (prompt, tool calls, output, latency, cost) is recorded.
3. Evaluate — automated evals score the run against your criteria.
4. Optimize — low-scoring prompts or model choices get revised and re-tested.
5. Promote — the version that scores best becomes the one serving traffic.

Break any link and the loop stalls: routing without tracing means you can't debug; tracing without evals means you're reading logs by hand; evals without optimization means you know you're worse but not what to change.

Platform vs. adjacent tools

The practical difference is integration cost. Separate tools can each be best-in-class, but you own the glue between them — and the glue is where version mismatches and missing context usually appear.

Signals you've outgrown ad-hoc scripts

• You call more than one model provider, or you want the option to.
• Agents are running in production and you can't reconstruct why a specific run failed.
• Quality regressions reach users before you notice them.
• You're comparing prompt versions by eyeballing outputs.
• Multiple engineers are changing prompts and no one knows which version is live.

Any two of these together usually justify evaluating a platform.

What to check when comparing options

• Provider coverage and routing control — can you route by cost, latency, or task, and fail over automatically?
• Trace depth — full agent trajectories or just LLM calls?
• Eval flexibility — custom criteria, offline datasets, and live-traffic scoring, or only fixed scorers?
• Closed loop — do eval results actually drive which version ships, or is that manual?
• Pricing model — Respan publishes a pricing page at respan.ai/pricing; check whether cost scales with requests, traces, or seats, since that determines what gets expensive as you grow. The available material doesn't specify the pricing structure, so read the page directly rather than assuming.

The right platform is the one whose loop you'll actually close. If your team won't act on eval results, the eval feature is decoration — pick based on the step you're genuinely ready to run.

LLM evals are structured tests that score model or agent outputs against defined criteria so you can compare versions and catch quality regressions. Use them when you need to choose between prompts, models, or agent configurations, or when you want an automated signal that quality changed after a deployment. They are not the same as observability: observability tells you what happened on live traffic, while evals tell you how good the output was on a controlled set of cases.

Evals vs. observability vs. monitoring

These three are often conflated, but they answer different questions and run at different times.

A practical split: use observability to see real agent behavior and collect interesting production cases, then promote those cases into your eval set. Respan's positioning reflects this pairing — it describes itself as an AI router with built-in observability and automated evals, and its page headings cover routing across models, shipping the best-scoring version, knowing when things change, and seeing what agents did. That is the loop: route, observe, evaluate, ship.

Core evaluation methods

There is no single metric that covers everything. Most teams combine two or three of these.

Automated metrics

Deterministic checks that need no model judgment. Good for tasks with a verifiable answer.

• Exact match or normalized match for classification, extraction, or short answers.
• Contains/regex checks for required fields, formats, or forbidden strings.
• JSON/schema validity for structured output.
• Retrieval metrics (precision, recall, hit rate) when the task depends on fetched context.

These are cheap, fast, and reproducible. They fail on open-ended generation, where many correct answers exist.

LLM-as-judge

A model scores outputs against a rubric. Useful for summarization, tone, helpfulness, and other qualities that resist exact matching.

• Write the rubric explicitly: what counts as pass, what counts as fail, and how ties are handled.
• Give the judge the input, the output, and the reference answer if you have one.
• Ask for a score plus a short justification so you can audit disagreements.
• Watch for known biases: judges tend to prefer longer answers and their own model family's style. Randomize or swap answer order when comparing two outputs.

Human review

People label a sample of outputs, usually for calibration rather than full coverage. Use it to validate that your automated metric actually tracks what you care about. If the judge and humans disagree often, fix the rubric before trusting the judge at scale.

Building a test set

The eval set is the part most teams underinvest in. A small, well-chosen set beats a large, noisy one.

1. Start from real usage. Pull representative inputs from production traces rather than inventing them.
2. Cover the edges. Include ambiguous requests, multi-step tasks, and cases where the correct behavior is to refuse or ask a clarifying question.
3. Write expected outputs or criteria. For open-ended tasks, criteria ("must cite the source document") work better than a single gold answer.
4. Label each case with the failure mode it targets, so a score drop tells you what broke, not just that something broke.
5. Keep a held-out slice. Do not tune prompts against every case you own, or you will overfit.

Running an eval to compare versions

The workflow is the same whether you compare prompts, models, or agent versions.

1. Freeze the variables. Change one thing at a time — prompt, model, or agent config — and hold the rest constant.
2. Run every version on the same test set. Same inputs, same order, same judge settings.
3. Score each run with your chosen metrics and record per-case results, not just the average.
4. Compare. Look at the aggregate score and the per-case diff. A version that wins on average but breaks a critical case is usually not the one to ship.
5. Ship the best-scoring version, then keep the eval set in CI so future changes are checked against it.

Respan frames the outcome as "ship the version that scores best," which is the decision this workflow is meant to support.

Catching regressions

A regression is a quality drop caused by a change that looked unrelated. Common triggers: a model version bump, a prompt edit, a retrieval change, or a new tool in an agent.

• Re-run the eval set on every prompt, model, or agent change, ideally automatically.
• Alert on score drops beyond a threshold you set, not on every fluctuation.
• Track per-case pass/fail over time so you can see which categories degrade.
• Keep a small "canary" subset of critical cases that must never fail.

Common pitfalls

• Overfitting to a small eval set. If you iterate against the same 20 cases, you optimize for those cases, not the task. Hold out a slice and refresh cases from production.
• Relying on a single metric. One number hides tradeoffs. Track at least one correctness metric and one quality metric.
• Trusting an unvalidated judge. Check the judge against human labels before scaling it.
• Eval sets that drift from reality. Production inputs change; a set built six months ago may no longer represent your users.
• No baseline. Without a recorded baseline score, you cannot tell whether a change helped or hurt.

Where to start

If you are setting this up for the first time: pick one task, collect 30–50 real inputs, define pass/fail criteria, and run a single automated metric plus an LLM judge. Compare two prompt versions and record per-case results. Once that loop works, add it to CI and grow the set from production traces.

LLM observability is the practice of capturing, tracing, and evaluating every model and agent run so you can see what your application actually did — not just whether it was up. It matters when you run multi-step agents, route across multiple models, or ship prompt changes regularly, because uptime and error-rate dashboards won't tell you why an answer was wrong or which step burned the tokens. If your application is a single stateless prompt with no tool calls, basic logging may be enough; the more steps, models, and tools involved, the more you need structured traces and evals.

Observability vs. traditional monitoring

Traditional monitoring answers "is the service responding?" LLM observability answers "what happened inside this run, and was the output any good?"

The two are complementary. You still want uptime and latency alerting; observability adds the layer that explains output quality.

Core signals to capture

Instrument at the level of the individual model call and the surrounding workflow step, then roll up. The signals worth capturing on every run:

• Prompts and completions — the actual input sent and output returned, including system messages and any retrieved context.
• Latency — per call and per span, so you can separate model time from tool time from your own code.
• Token cost — input and output tokens per call, attributed to a model, a feature, or a user.
• Tool calls — which tools were invoked, with what arguments, and what came back.
• Errors — provider errors, timeouts, malformed tool arguments, and schema validation failures.

Capturing prompts and completions is what makes debugging possible; capturing cost and latency per span is what makes optimization possible.

How tracing connects multi-step agent workflows

A single trace represents one end-to-end run. Inside it, each step is a span: a model call, a retrieval, a tool invocation, a routing decision. Spans nest, so a parent span for "answer user question" can contain child spans for "retrieve documents," "call model," and "call calculator."

That structure is what lets you trace a failure to a specific step. If the final answer is wrong, the trace shows whether retrieval returned irrelevant documents, whether the model ignored them, or whether a tool returned an error that the agent silently swallowed. Without spans, you only see the bad final output and have to guess.

For agents that route across multiple models — for example, sending easy requests to a cheaper model and hard ones to a stronger model — the trace should record which model handled each call. Respan describes itself as an AI router with built-in observability and automated evals, which is the pattern to look for: routing decisions and observability data captured in the same place, so you can compare cost and quality across the models you route between.

Turning traces into quality signals with evals

Traces tell you what happened; evals tell you whether it was good. Automated evals score runs — against reference answers, rubrics, or model-based judges — and attach those scores to the trace. Over time, that gives you a quality trend line you can compare across prompt versions, model versions, and releases.

The practical loop:

1. Capture traces for every run.
2. Score a sample (or all) of them with automated evals.
3. Compare scores across prompt or model changes.
4. Ship the version that scores best, and keep monitoring after release.

This is why "ship the version that scores best" and "know the moment things change" belong together: evals give you the comparison, and continuous monitoring tells you when a previously good version starts degrading — for example, after a provider silently updates a model.

What to check when observability data looks wrong or missing

If traces are incomplete or scores look off, work through these common causes:

• Uninstrumented calls. A code path that calls a model directly, bypassing your gateway or SDK wrapper, produces no span. Audit for direct provider clients.
• Dropped spans. Async or background work that finishes after the parent run closes can lose its span. Check that spans are flushed before the trace is finalized.
• Missing context. If prompts are logged but retrieved documents aren't, you can't tell whether a bad answer came from bad retrieval. Instrument retrieval as its own span.
• Sampling gaps. If you sample traces, make sure eval scores and cost totals account for the sampling rate rather than reporting sampled numbers as totals.
• Clock and ordering issues. Out-of-order timestamps make latency attribution misleading; verify timestamps come from a consistent source.

Choosing what to instrument first

Start with the signals that map to decisions you actually make. If you're optimizing cost, instrument tokens and model routing per call. If you're debugging quality, instrument prompts, completions, retrieval, and tool calls, and add evals. If you're managing reliability, instrument errors and latency per span. A platform that combines routing, tracing, and automated evals — as Respan positions itself — reduces the work of stitching those layers together, but the signals above are what you need regardless of which tool provides them.

An LLM gateway is a single routing layer that sits between your application and multiple model providers, exposing one API while handling failover, load balancing, rate limits, caching, and usage tracking behind it. You need one when you call more than one provider, when a single provider's outage or rate limit would break your product, or when you want centralized control over keys and spend. You do not need one if you call a single model at low volume and have no reliability or cost-visibility problem to solve.

What an LLM gateway actually is

A gateway is infrastructure, not an intelligence layer. It receives a request in a normalized format, decides which provider or model should serve it, forwards the call, and returns a normalized response. Everything the gateway does happens on the request path; it does not judge whether the answer was good.

That distinction matters because gateways are often bundled with adjacent tools. Respan, for example, describes itself as "the AI router with built-in observability & automated evals" — a router plus monitoring plus evaluation in one product. You can also run a gateway alone and keep observability and evals elsewhere. The routing function is the defining part.

Core capabilities to expect

Respan's own headings frame the same set of concerns: "Reach every model, and stay up" (provider coverage plus resilience), "Ship the version that scores best" (routing tied to evaluation results), "Know the moment things change" (monitoring), and "See exactly what your agents did" (trace-level visibility into agent runs).

Benefits and trade-offs

The upside is provider flexibility and resilience. If one model is down, rate-limited, or suddenly more expensive, a gateway lets you shift traffic without shipping a new client. Centralized key management also means credentials live in one place instead of scattered across services.

The costs are real and worth naming:

• Added latency. Every request takes an extra network hop. For latency-sensitive paths, measure this before committing.
• Extra infrastructure. A gateway is another service to deploy, scale, and monitor.
• A new point of failure. If the gateway goes down and you have no bypass, it takes your whole model layer with it. Plan a direct-to-provider fallback for critical paths.
• Configuration surface. Routing rules, fallback order, and cache policies are things someone has to own.

How a gateway fits with observability and evals

These are three different jobs that are easy to conflate:

• The gateway routes traffic. It decides where a request goes and keeps it flowing.
• Observability measures what happened. Latency, error rates, token usage, and full traces of agent runs.
• Evals measure whether output was good. Scoring responses against criteria so you know which prompt or model version actually performs better.

A gateway improves reliability and control; it does not tell you whether your answers improved. If your problem is "our outputs got worse after a prompt change," a gateway will not solve it — evals will. If your problem is "one provider's outage took us down for an hour," the gateway is the fix. Products like Respan combine all three, which reduces integration work but also means you are adopting a platform, not just a proxy.

When adopting a gateway is worth the overhead

Adopt one when at least one of these is true:

1. You call two or more providers and want to switch between them without code changes.
2. A provider outage or rate limit would break your product, and you need automatic failover.
3. Keys and spend are scattered across teams or services and you need central control.
4. You want to route by evaluation results — sending traffic to whichever model version scores best.

Stay with direct provider calls when you use a single model, traffic is low, and you have no reliability or cost-visibility pain. Adding a hop to solve a problem you do not have is a net loss.

Checklist for choosing a gateway

Evaluate candidates on the same dimensions:

• Provider and model coverage. Does it support every provider you use today, and the ones you might add?
• Streaming support. If your app streams tokens, confirm the gateway passes streams through without buffering them.
• Latency overhead. Test it on your own traffic, not a vendor benchmark.
• Failover behavior. What triggers a fallback, how fast, and can you define the order?
• Self-hosting. If data residency or control matters, can you run it yourself?
• Pricing model. Check the vendor's pricing page for how you are charged — per request, per token, or per seat — and whether self-hosting changes it.
• Bypass path. Can you call providers directly if the gateway fails?

Answer those seven and you will know whether a given gateway fits, and whether the trade-off is worth it for your traffic.