Failover & fallbacks

Providers have bad minutes. INFRO absorbs them with two layers of resilience: provider failover, which automatically retries the same model on a different provider, and model fallbacks, an ordered list of alternate models you set per request. Both operate strictly before the first token.

Provider failover is same-model and always on — you never configure it. Fallbacks are cross-model and opt-in via the fallbacks array. Either way, the response reports exactly what served the request and what it cost.

Provider failover

Most catalog models are served by multiple providers. When the chosen provider returns a 429 or 5xx, or times out before the first token, INFRO re-routes to the next eligible provider for the same model. Your client sees nothing — no error, no configuration, just slightly higher time to first token.

Failover respects your routing configuration: providers in routing.providers.deny are never tried, routing.providers.allow restricts the pool, and routing.regions pins the whole chain to the regions you list. If you attach provider keys via BYOK, your key is preferred for that provider, and failover to other providers still applies when it errors.

Request-level errors — 400, 401, 402, 404 — return immediately without failover. A different provider would not change the outcome, so INFRO doesn't waste latency trying.

Model fallbacks

Fallbacks cover the case where an entire model is unavailable — every provider serving it is erroring or rate-limited. Pass a fallbacks array of model IDs and INFRO tries them in order, each with the same automatic provider failover as the primary.

fallbacksstring[]

Ordered list of model IDs to try once every provider of the primary model has failed before the first token. Each entry gets full provider failover of its own. Top-level request body field — pass it via extra_body in the OpenAI Python SDK, or directly in the request body in TypeScript.

curl https://api.infro.io/v1/chat/completions \
  -H "Authorization: Bearer $INFRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "moonshot/kimi-k2",
    "messages": [
      {"role": "user", "content": "Summarize this changelog in three bullets."}
    ],
    "fallbacks": ["zai/glm-4.6"]
  }'

Request lifecycle

1. INFRO selects a provider for the primary model according to your routing policy.
2. If that provider returns 429 or 5xx, or times out before the first token, INFRO retries the next eligible provider for the same model.
3. When every eligible provider for the primary model has failed, INFRO moves to the first entry in fallbacks and repeats the same provider failover — then the second entry, and so on.
4. When the chain is exhausted, the request fails: 502 upstream_error if providers returned errors, or 503 no_available_provider if no provider was eligible to try.

Both terminal statuses are retryable — use exponential backoff with jitter, the same as 408 and 429. See Errors for the full retry matrix.

Reading the response

The response never hides what happened. Three fields report the truth:

modelstring

The model that actually served the request. With fallbacks configured, compare it against the model you asked for to detect a fallback serve.

providerstring

The provider that served the request, after any failover.

usage.costnumber

The exact USD charged, at the served model's per-token rate.

{
  "id": "gen-8fJq3k",
  "model": "zai/glm-4.6",
  "provider": "novita",
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "- ..." },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 214,
    "completion_tokens": 96,
    "total_tokens": 310,
    "cost": 0.0002846
  }
}

Streaming

With stream: true, both layers apply in full before the first token — the only symptom of a failover chain is a longer wait for the first chunk. Once tokens flow, INFRO cannot restart your request on a different provider without silently replaying content, so it doesn't.

If a provider drops mid-stream, the stream surfaces an error chunk instead of data: [DONE]. Treat it as retryable: re-issue the request, and the full failover and fallback chain applies fresh. If you render partial output to users, be ready to discard it on retry. Chunk format and stream_options are covered in Streaming.

Choosing fallback models

A fallback only helps if its response is one your application can actually use. Pick fallbacks the way you would pick a substitute, not just a survivor:

• Match capabilities. If the primary route uses tool calling, structured outputs, or vision, every fallback must list the same entries in its capabilities array — check GET /v1/models or browse the catalog. A fallback that can't emit tool_calls breaks your route in a new way instead of saving it.
• Match context length. A fallback with a smaller window turns a request that fits today into a 400 during an outage — exactly when you can least afford it.
• Match tier and output style. moonshot/kimi-k2 falling back to zai/glm-4.6 works because they are peers: similar quality, similar formats, both tool-capable. Pairing a frontier model with a budget one is a different decision entirely.
• Keep the chain short. One or two entries. Each exhausted model adds latency before a real outage surfaces, and a long chain usually means the route's requirements aren't pinned down.

Fallbacks compose with everything else on the request: routing constraints apply to every model in the chain, and "policy": "cheapest" picks the provider within each model, not across models. See Routing for how the two interact, and Chat completions for the full request schema.