Prompt caching

Most providers cache the repeated portion of a prompt and bill it at a reduced rate when a later request reuses it. INFRO passes this through automatically — OpenAI, Anthropic, DeepSeek, and Google style caching all work with no INFRO-side configuration — and the discount shows up in the exact usage.cost on every response.

There is nothing to enable and no INFRO-specific cache flag. If your requests share a stable prefix and land on a provider that caches, you pay less. This page covers structuring prompts so caches actually hit, how Anthropic's explicit cache_control blocks are handled, and how to confirm the savings.

How it works

Caching happens at the provider, not INFRO. When a provider recognizes the start of an incoming request as one it processed recently, it skips recomputing that prefix and bills those input tokens at a discount. INFRO forwards your request unchanged, meters what the provider actually charged, and reports it as usage.cost — see Chat completions for the full response shape.

Providers set these discounts and change them over time — treat the numbers above as ballpark. The authoritative figure is usage.cost on the actual response. Per-model base rates come from GET /v1/models and the models page; see Models for the catalog format.

Caches live with individual providers, so hits depend on consecutive requests reaching the same provider. That is the common case, but if hit rates matter for a high-volume workload, pin traffic to specific providers with routing.providers.allow — see Routing.

Structure prompts for cache hits

Provider caches match on the exact token prefix of a request. Only the portion identical to a recent request — starting from the very beginning — can hit the cache. The practical rule: stable content first, volatile content last.

1. System prompt — identical on every request. No timestamps, request IDs, or user names interpolated into it.
2. Tool definitions — same tools, in the same order, with the same descriptions.
3. Conversation history — append-only, so earlier turns keep matching the cached prefix as the conversation grows.
4. The newest user message — the volatile part goes last.

Anthropic cache_control

Anthropic models use explicit cache breakpoints rather than fully automatic caching. Mark where the cacheable prefix ends with a cache_control block on a content part; INFRO passes it through to Anthropic unchanged — no translation, no stripping. Anthropic allows up to four breakpoints per request; one at the end of your stable prefix is usually enough.

cache_controlobject

Attach to a content part to mark the end of a cacheable prefix. Meaningful on anthropic/... models only — providers with automatic caching ignore it.

cache_control.typestringrequired

The cache type. Use "ephemeral".

{
  "model": "anthropic/claude-sonnet-5",
  "messages": [
    {
      "role": "system",
      "content": [
        {
          "type": "text",
          "text": "You are a support agent for Acme. <several thousand tokens of policy>",
          "cache_control": { "type": "ephemeral" }
        }
      ]
    },
    { "role": "user", "content": "How do I reset my password?" }
  ]
}

The first request writes the cache; later requests within its lifetime that share the exact prefix read from it at the discounted rate. Everything up to and including the marked block is cached — content after it is processed normally.

Verify caching is working

The reliable signal is usage.cost. Send the same request twice in quick succession and compare: with a prompt long enough to cache, the second call should be noticeably cheaper. Token counts do not move — cached tokens still appear in prompt_tokens — so watch cost, not tokens.

# request.json holds a long, stable system prompt plus a user message
for i in 1 2; do
  curl -s https://api.infro.io/v1/chat/completions \
    -H "Authorization: Bearer $INFRO_API_KEY" \
    -H "Content-Type: application/json" \
    -d @request.json | jq '.usage.cost'
done

On streaming requests, pass stream_options with {"include_usage": true} to receive a final chunk whose usage includes prompt_tokens, completion_tokens, and cost — see Streaming. If the second call is not cheaper, check that the prefix is byte-for-byte identical and that both requests hit the same provider — the top-level provider field on the response tells you.

What savings to expect

Cached input generally costs roughly 50–90% less than uncached input, varying by provider and model. Output tokens are always billed at the full rate — the discount applies only to the cached portion of the prompt.

• Caches are short-lived — typically minutes, depending on the provider. Steady traffic benefits most; a request arriving long after the previous one usually pays full price.
• Short prompts may never cache. Providers enforce minimum prefix lengths, commonly around 1,024 tokens.
• Some providers bill the initial cache write slightly above the normal input rate, so the first request can cost marginally more than an uncached one.
• Caching is provider-side compute reuse, independent of INFRO logging: requests sent with logging: false still get cache discounts. See Privacy.