The first provider outage you sleep through is the one that convinces you. Most teams running LLMs in production hit the same sequence: the model that was clearly best in March gets leapfrogged by June, a rate limit throttles a launch, an outage takes the product down for forty minutes, and finance asks why the inference bill tripled. None of these are model problems. They're infrastructure problems, and the piece of infrastructure that addresses all four is an LLM gateway.
What is an LLM gateway?
An LLM gateway — also sold as an AI gateway or LLM router; the terms overlap almost completely — is a proxy layer between your application and every model provider you use. Your code speaks one API, almost always the OpenAI chat completions format, which won the standards war by default. The gateway handles the rest: translating requests into each provider's wire format, picking which provider serves each call, retrying failures, caching repeats, metering spend, and logging what happened.
The closest analogy is a load balancer, but model-aware. A load balancer knows which backends are healthy. A good gateway also knows what each model costs per million tokens, its context window, how its tool-calling format deviates from spec, and which provider is serving it fastest right now. That's knowledge you don't want hand-maintained across your application code.
The multi-model problem
You can run production on a single provider, and plenty of teams do — right up until one of four things happens.
- Vendor lock-in. Every provider SDK you integrate is a small commitment; every prompt tuned to one model's quirks is a bigger one. When Kimi K2 or GLM-4.6 starts matching your incumbent on your actual workload at a fifth the price — routine in 2026 — switching should be a config change, not a migration project.
- Rate limits. Provider limits are negotiated per account and don't care about your launch. One traffic spike into a 429 wall and your product degrades exactly when it's most visible. Spreading load across two or three providers serving the same open-weight model is the boring, effective fix.
- Outages. Every major provider has had multi-hour incidents. Status pages are green until they aren't. If your fallback plan is "wait", your uptime is capped at one vendor's uptime.
- Cost sprawl. Five API keys, five dashboards, five invoices, and no single answer to "what did feature X cost last week?" The spread between comparable models is wide — Claude Opus 5 lists at $5 per million input tokens and $25 out; DeepSeek V3.2 at roughly $0.28 and $0.42, list prices at the time of writing — so routing the easy 80% of traffic to a cheaper model is often the biggest single lever on your bill. You can't pull it if every call site is hard-wired.
The common thread: model choice is a runtime decision that most codebases treat as a compile-time one. A gateway moves it back to runtime.
Core gateway capabilities
Vendors bundle these differently, but a real gateway — as opposed to a thin proxy — does six things.
| Capability | What it does | Why it matters in production |
|---|---|---|
| Unified API | One OpenAI-compatible endpoint for every model | Swapping models means changing a string, not an SDK |
| Smart routing | Picks a provider per request on price, latency, and health | 5–20x price gaps between comparable models become exploitable |
| Failover | Retries on another provider on 429/5xx/timeout | Uptime is no longer bounded by one vendor |
| Caching | Serves repeated requests from a stored response | Duplicate prompts and retries cost nothing the second time |
| Spend controls | Budgets and limits per key, team, or feature | A leaked key or looping agent hits a wall, not your card |
| Observability | Logs tokens, latency, cost, and provider per request | "What did this feature cost?" gets a query, not a spreadsheet |
The unified API is what makes the rest cheap to adopt. If you already use the OpenAI SDK, pointing it at a gateway is a two-line change:
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.infro.io/v1", # was https://api.openai.com/v1
api_key=os.environ["INFRO_API_KEY"],
)
response = client.chat.completions.create(
model="deepseek/deepseek-v3.2", # or gpt-5.1, claude-opus-5, glm-4.6 ...
messages=[{"role": "user", "content": "Classify this support ticket: ..."}],
)Streaming, tool calls, and structured outputs keep working because the gateway speaks the same protocol on both sides. The model string becomes the only place a vendor is named.
The life of a request
Between client.chat.completions.create() and the first token, a typical gateway does the following:
- Your app sends a standard chat completions request to the gateway endpoint, authenticated with a gateway key instead of a provider key.
- The gateway validates the key and checks policy: remaining spend budget, rate budget, allowed models.
- Cache check. An identical recent request returns the stored completion in single-digit milliseconds at zero cost.
- On a miss, the router picks a provider: who serves the requested model, current prices, recent latency and error rates, health.
- The request is translated into that provider's wire format — parameter names, tool-call schemas, streaming framing — and forwarded.
- On a 429, 5xx, or timeout, the gateway fails over to the next candidate: another provider serving the same model, or a configured fallback model.
- Tokens stream back as normal SSE chunks. The gateway relays rather than buffers, so streaming is preserved end to end.
- After completion, it records tokens in and out, computed cost, latency, and which provider served the call — the raw material for analytics and billing.
A gateway is an extra network hop. Well-run ones add roughly 20–50 ms to time-to-first-token — noise against generation times measured in seconds, but if you're streaming for voice or another latency-critical path, measure it on your own traffic.
Gateway vs direct provider calls
| Direct to providers | Through a gateway | |
|---|---|---|
| Integration | One SDK and auth flow per provider | One SDK, one key |
| Failover | You build and maintain it | Built in |
| Cost tracking | Per-provider dashboards, manual reconciliation | One ledger across all models |
| Latency | No extra hop | Typically +20–50 ms per request |
| Day-one provider features | Available immediately | Can lag until gateway support lands |
| Prompt data path | You and the provider | One more party in the path — read the retention policy |
Direct calls are the right answer more often than gateway vendors like to admit: one model, one provider, modest volume, no uptime SLO worth engineering for. If that's you, skip the hop. The calculus flips once you run more than one model, care about uptime, or want the price gap between Western flagships and the frontier-class open-weight tier — DeepSeek, Qwen, Kimi, GLM — that now matches them on many workloads at 5–20x lower list prices.
Build vs buy
Building tempts because the happy path is easy — a proxy forwarding chat completions to two providers is a weekend project. The long tail is the actual job: normalizing tool-call formats that every provider breaks differently, translating streaming frames, tracking provider health without flapping, maintaining a price table for 120-plus models that changes monthly, getting cache correctness right, and reconciling token counts against five invoices. Teams that build in-house tend to acquire a permanent part-time infrastructure owner nobody planned for.
Build if the gateway is your product, or if you have data-path requirements no vendor can meet — fully on-prem inference, strict residency guarantees. Otherwise it's undifferentiated heavy lifting, and the buy side of the ledger gets heavier every quarter as the catalog grows.
How to choose an LLM gateway
- Protocol depth, not just breadth. Streaming, tool calls, structured outputs, and vision should all work through the OpenAI-compatible surface. Test the corners, not the happy path.
- A catalog with open-weight models. Most of the 2026 cost win comes from DeepSeek, Qwen, Kimi, GLM, and MiniMax. A gateway fronting only the big Western labs leaves the arbitrage on the table.
- Published latency overhead. If a vendor won't state added p50 and p99, assume the numbers are bad.
- Configurable failover. You control fallback order and whether cross-model fallback is allowed — a silent downgrade from Opus 5 to a small model is a bug, not a feature.
- Honest pricing. Know where the money comes from: a per-token markup, a subscription, or volume pricing passed through below list.
- Per-key spend controls, so one leaked key or runaway agent loop can't drain a month's budget in an afternoon.
- An exit path. Standard log exports and the OpenAI protocol mean you can leave. A gateway that solves vendor lock-in by creating its own has missed the point.
Where INFRO fits
INFRO is our implementation of this category, so calibrate accordingly — but the design follows from the checklist above. It's an OpenAI-compatible gateway in front of 120-plus models: GPT-5.1, Claude Opus 5 and Sonnet 5, Gemini 3 Pro, and the full open-weight tier including DeepSeek V3.2, Qwen3-Max, Kimi K2, GLM-4.6, and Llama 4 — the model catalog has current prices for all of them. Routing sends each request to the most cost-efficient provider that's currently reliable, failover is automatic, and billing is one ledger. Because we buy inference in volume, most models are priced below the provider's list price, and the rest are shown at list; the pricing section shows live numbers rather than us asserting them here.
It won't be right for everyone. One model at low volume with no uptime target doesn't need a gateway yet. Where it earns its keep is the blend: one support-bot workload we modeled routed classification and extraction to DeepSeek V3.2 at roughly $0.28/$0.42 per million tokens and kept Opus 5 for the reasoning-heavy 10% of traffic. The blended bill came out around 70% lower than all-Opus, with no application change beyond the model string.
To check the numbers against your own workload, the quickstart is a base-URL swap that takes about five minutes. Run a slice of real traffic through it, compare the invoice, and decide with data.