Smart routing

Most models on INFRO are served by more than one provider. On every request, the router scores the healthy providers for your model and picks one — you send a single model ID, and INFRO handles selection, price differences, and failover behind the scenes.

The default policy is cheapest, so with zero configuration you pay the lowest available per-token rate. When you need a different tradeoff — lower latency, a specific provider, traffic pinned to a region — set the routing object on the request. This page covers scoring, the three policies, and every field of routing.

How providers are scored

The router keeps a live scorecard for every provider of every model, built from three inputs:

• Price — the per-token prompt and completion rates each provider charges for the model.
• Latency — rolling time-to-first-token and throughput, measured from real traffic per region.
• Health — live error rates, timeouts, and rate-limit pressure over a recent window.

Your policy sets the price/latency weighting; health acts as a filter on top. Providers with elevated error rates are deprioritized, and one that is hard down drops out of the candidate set until it recovers. Because scores come from live traffic rather than static benchmarks, routing adapts within minutes when a provider degrades.

Selection is also the first line of failover. If the chosen provider returns a 429 or 5xx, or times out before the first token, the router retries the next-ranked provider invisibly — the request fails with 502 upstream_error only after every provider has errored. Once streaming begins, a provider drop surfaces as an error chunk instead, and your client should retry. See Errors and Streaming.

Routing policies

cheapest ignores latency entirely. fastest pays whatever the quickest healthy provider charges. balanced starts from price but excludes providers whose recent latency is well off the pace, avoiding the occasional slow host without the full fastest premium. All three choose only among healthy providers — a cheap provider that is failing requests is no bargain, and the router treats it that way.

The routing object

routing is a top-level field in the POST /v1/chat/completions body. The official OpenAI SDKs don't know about it, so pass it via extra_body in Python or as an untyped extra field in TypeScript. See Chat completions for the full request schema.

policystring

"cheapest" (default), "fastest", or "balanced". Sets the price/latency weighting used to rank providers.

providers.allowstring[]

Restrict routing to these provider slugs. Providers not listed are never used for this request.

providers.denystring[]

Never route to these providers. Applied after allow — a provider on both lists is denied. Also overrides BYOK preference for that provider.

regionsstring[]

Any of "us", "eu", "ap". Only provider deployments in the listed datacenter regions are considered.

Provider slugs are lowercase vendor names — the same strings the response's provider field reports (for example openai, deepseek). The example below runs an open-weight model at the fastest available host, excludes the first-party API, and keeps traffic in the US and EU:

curl https://api.infro.io/v1/chat/completions \
  -H "Authorization: Bearer $INFRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek/deepseek-v3.2",
    "messages": [{"role": "user", "content": "Summarize this contract clause: ..."}],
    "routing": {
      "policy": "fastest",
      "providers": {"deny": ["deepseek"]},
      "regions": ["us", "eu"]
    }
  }'

If your filters leave no healthy provider — an allow list of one provider that is down, or a region where the model isn't hosted — the request fails with 503 no_available_provider rather than silently relaxing your constraints. The router never substitutes a different model; switching models is the job of fallbacks.

Region pinning

Set routing.regions to keep a request inside specific datacenter regions: us, eu, or ap. Scoring works as before, but only provider deployments in those regions are candidates — useful for data-residency requirements or for keeping latency predictable for users in one geography.

{ "routing": { "policy": "balanced", "regions": ["eu"] } }

Pinning is per request, so you can pin EU traffic for European tenants while leaving everything else unrestricted. Independently of pinning, open-weight models from Chinese labs (DeepSeek, Qwen, Moonshot, Z.ai, MiniMax) are served from non-China datacenter regions by default. See Privacy for the full data-handling picture.

Which provider served the request

Every response carries a top-level provider field with the slug of the provider that served it, and usage.cost reports the exact USD charged. When fallbacks trigger, the response's model field reports the model that actually ran — model plus provider fully identifies where your tokens went.

{
  "id": "chatcmpl-8fJxT2",
  "object": "chat.completion",
  "model": "anthropic/claude-sonnet-5",
  "provider": "anthropic",
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "..." },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 412,
    "completion_tokens": 96,
    "total_tokens": 508,
    "cost": 0.002676
  }
}

In the OpenAI Python SDK, INFRO extensions land in model_extra — read it as completion.model_extra["provider"]. Log it alongside your own request IDs: when latency or output quality shifts, the first question is whether the provider mix changed, and this field answers it.

Routing with BYOK and fallbacks

If you have attached your own provider key (BYOK), the router prefers your key whenever it considers that provider — the provider bills you directly and INFRO charges 5% of INFRO-rate equivalent. Failover still applies: if requests on your key start rate-limiting or erroring, the router moves on as with any provider. Putting a provider in providers.deny denies it entirely, your own key included.

Routing and fallbacks operate at different levels: routing selects among providers of one model, while fallbacks switches to a different model, and only after every provider of the primary has failed. The same routing object then governs provider selection for each fallback model in turn. Both act before the first token — nothing reroutes mid-stream.

The full catalog, including per-model pricing and capabilities, is available at /models and programmatically via GET /v1/models.