What Is an LLM Gateway and When Do You Need One?
An LLM gateway is a single routing layer that sits between your application and multiple model providers, exposing one API while handling failover, load balancing, rate limits, caching, and usage tracking behind it. You need one when you call more than one provider, when a single provider's outage or rate limit would break your product, or when you want centralized control over keys and spend. You do not need one if you call a single model at low volume and have no reliability or cost-visibility problem to solve.
What an LLM gateway actually is
A gateway is infrastructure, not an intelligence layer. It receives a request in a normalized format, decides which provider or model should serve it, forwards the call, and returns a normalized response. Everything the gateway does happens on the request path; it does not judge whether the answer was good.
That distinction matters because gateways are often bundled with adjacent tools. Respan, for example, describes itself as "the AI router with built-in observability & automated evals" — a router plus monitoring plus evaluation in one product. You can also run a gateway alone and keep observability and evals elsewhere. The routing function is the defining part.
Core capabilities to expect
| Capability | What it does | Why it matters |
|---|---|---|
| Unified API | One request format across providers | Swap or add models without rewriting client code |
| Failover and retries | Re-sends failed calls to another provider or model | Keeps the product up when one provider degrades |
| Load balancing | Spreads traffic across models or keys | Avoids hitting per-key rate limits |
| Rate limiting | Caps request volume per caller or key | Protects upstream quotas and downstream budget |
| Caching | Returns stored responses for repeated inputs | Cuts cost and latency on duplicate traffic |
| Cost and usage tracking | Attributes spend to keys, teams, or features | Makes multi-provider spend legible |
Respan's own headings frame the same set of concerns: "Reach every model, and stay up" (provider coverage plus resilience), "Ship the version that scores best" (routing tied to evaluation results), "Know the moment things change" (monitoring), and "See exactly what your agents did" (trace-level visibility into agent runs).
Benefits and trade-offs
The upside is provider flexibility and resilience. If one model is down, rate-limited, or suddenly more expensive, a gateway lets you shift traffic without shipping a new client. Centralized key management also means credentials live in one place instead of scattered across services.
The costs are real and worth naming:
- Added latency. Every request takes an extra network hop. For latency-sensitive paths, measure this before committing.
- Extra infrastructure. A gateway is another service to deploy, scale, and monitor.
- A new point of failure. If the gateway goes down and you have no bypass, it takes your whole model layer with it. Plan a direct-to-provider fallback for critical paths.
- Configuration surface. Routing rules, fallback order, and cache policies are things someone has to own.
How a gateway fits with observability and evals
These are three different jobs that are easy to conflate:
- The gateway routes traffic. It decides where a request goes and keeps it flowing.
- Observability measures what happened. Latency, error rates, token usage, and full traces of agent runs.
- Evals measure whether output was good. Scoring responses against criteria so you know which prompt or model version actually performs better.
A gateway improves reliability and control; it does not tell you whether your answers improved. If your problem is "our outputs got worse after a prompt change," a gateway will not solve it — evals will. If your problem is "one provider's outage took us down for an hour," the gateway is the fix. Products like Respan combine all three, which reduces integration work but also means you are adopting a platform, not just a proxy.
When adopting a gateway is worth the overhead
Adopt one when at least one of these is true:
- You call two or more providers and want to switch between them without code changes.
- A provider outage or rate limit would break your product, and you need automatic failover.
- Keys and spend are scattered across teams or services and you need central control.
- You want to route by evaluation results — sending traffic to whichever model version scores best.
Stay with direct provider calls when you use a single model, traffic is low, and you have no reliability or cost-visibility pain. Adding a hop to solve a problem you do not have is a net loss.
Checklist for choosing a gateway
Evaluate candidates on the same dimensions:
- Provider and model coverage. Does it support every provider you use today, and the ones you might add?
- Streaming support. If your app streams tokens, confirm the gateway passes streams through without buffering them.
- Latency overhead. Test it on your own traffic, not a vendor benchmark.
- Failover behavior. What triggers a fallback, how fast, and can you define the order?
- Self-hosting. If data residency or control matters, can you run it yourself?
- Pricing model. Check the vendor's pricing page for how you are charged — per request, per token, or per seat — and whether self-hosting changes it.
- Bypass path. Can you call providers directly if the gateway fails?
Answer those seven and you will know whether a given gateway fits, and whether the trade-off is worth it for your traffic.