Caching

MeshAPI has two independent caching layers. They solve different problems and can be used together:

Layer	What it reuses	Where it lives	Who pays
Gateway response cache	The entire response to an identical request	MeshAPI (Redis)	Free by default — the upstream call never happens
Provider prompt caching	The shared prefix of a prompt across different requests	The provider (Claude, Amazon Nova)	On Claude: reads cost 0.10× input, writes 1.25× — pays off as soon as a prefix is reused

The gateway cache is on by default and needs no code changes. Provider prompt caching is opt-in per request via cache_control markers.

Gateway response cache

When a request is byte-for-byte equivalent to one served recently, MeshAPI returns the stored response and never calls the provider. The result is a response in single-digit milliseconds at no token cost. The response cache covers POST /v1/chat/completions and POST /v1/messages. It also covers POST /v1/responses in two cases:

the request sets store: false;
the model has no native Responses API, so MeshAPI translates the request.

A stored Responses request on a model with a native Responses API is never cached; store: true is the default. A Responses request is matched on its whole body, apart from stream, user and service_tier, rather than on the fields below. A hit replays the answer in the Responses format.

What counts as “identical”

The cache key is a SHA-256 hash over exactly these fields:

model
the resolved upstream provider
messages
max_tokens
stop
response_format
service_tier, when the request selected one — see Service Tiers
each of temperature, seed, top_p, frequency_penalty, presence_penalty, reasoning_effort and reasoning, when the request sets it (unset is its own value, not 0)
the parameters the serving model could not accept and Mesh withheld, if any

Anything not in that list does not affect the cache key. Two things follow that surprise people:

stream is not part of the key. A streaming request can be served from a response originally stored by a non-streaming one; MeshAPI replays it as normal SSE chunks. Your streaming client sees no difference.
user is not part of the key. Passing a different user value does not give you a different cache entry.

When caching applies

A request is eligible only if all of these hold:

Condition	Why
`temperature` is `0` or omitted	Any non-zero temperature means non-deterministic output, which must not be reused. An omitted temperature is still cached, but under its own entry, separate from `temperature: 0`, because the provider sampled it at its own default (usually `1.0`).
No `tools` in the request	A tool call depends on live state; replaying one would be wrong.
Not an image-generation model	Image output is not stored in the response cache.
No memory injected	A request whose `x-mem-id` memories were injected is personalized to one user. See Memory. An `x-mem-id` that injects nothing doesn’t bypass the cache.
Response is under 64 KB serialized	Larger responses are served normally but never written to the cache.

The temperature rule is the one that catches people out. If you send "temperature": 0.7, you get no caching at all, no matter how many times you repeat the identical prompt. Deterministic workloads (classification, extraction, structured output) should send temperature: 0 explicitly.

Detecting a cache hit

A response served from the cache carries:

X-Cache: HIT

There is no X-Cache: MISS header. The header is present only on a hit — treat its absence as a miss rather than checking for a MISS value.

The header is the reliable per-request signal: count X-Cache: HIT responses to measure your hit rate. The per-request Usage API events don’t carry the hit marker, so you can’t filter on it there.

Turning it off

Caching is enabled by default. You can opt out per request in two ways:

Request header
Request body

curl https://api.meshapi.ai/v1/chat/completions \
  -H "Authorization: Bearer <YOUR_RSK_KEY>" \
  -H "Content-Type: application/json" \
  -H "X-Mesh-Cache: no-store" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [{ "role": "user", "content": "What time is it in Tokyo?" }]
  }'

curl https://api.meshapi.ai/v1/chat/completions \
  -H "Authorization: Bearer <YOUR_RSK_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [{ "role": "user", "content": "What time is it in Tokyo?" }],
    "cache": false
  }'

Both skip the cache read and the cache write for that request.

"cache": true does not force caching on. The flag is deliberately asymmetric: false is an unconditional opt-out, but true only means “cache if otherwise permitted” — it still respects your key’s settings and every eligibility rule above. There is no way to cache a temperature: 0.9 request.

To disable caching for every request on a key, set it on the key itself rather than per request — see the Account Configuration Checklist.

Cache isolation

By default each account gets its own cache namespace: your responses are never served to another tenant, even for an identical prompt. Accounts may opt into a shared namespace, where entries are keyed by content hash alone and hit rates are higher because the cache is warmed by all participants. Shared caching is off unless you ask for it.

Freshness

Entries live for 24 hours. There is no explicit invalidation API — if you need a guaranteed-fresh answer, send X-Mesh-Cache: no-store.

Provider prompt caching

Provider prompt caching is a different mechanism: instead of reusing a whole response, the provider stores the processed prefix of your prompt and charges less for it on subsequent requests. It pays off when you send a large, stable block — a long system prompt, a document, a few-shot set — followed by a small varying question. Unlike the gateway cache, this one still calls the provider and still generates a fresh completion. It reduces the cost of the input tokens, not the call.

This one is not free money. Reading a cached prefix costs 10% of the input rate, but creating it costs 125% — so a prefix that is written and never read costs you more than not caching at all. See What it costs before turning it on across a workload.

Marking a prefix

Add a cache_control marker to the content you want cached. MeshAPI forwards it to providers that support server-side prompt caching.

curl https://api.meshapi.ai/v1/chat/completions \
  -H "Authorization: Bearer <YOUR_RSK_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-sonnet-5",
    "messages": [
      {
        "role": "system",
        "content": [
          {
            "type": "text",
            "text": "<a long, stable instruction block or document>",
            "cache_control": { "type": "ephemeral" }
          }
        ]
      },
      { "role": "user", "content": "Summarize section 4." }
    ]
  }'

Each marker defines a breakpoint that caches everything before it. Put markers at the end of the stable region, immediately before the part that changes between requests.

Where markers are honoured

Whether cache_control reaches the provider depends on which provider serves the model, not on the model name alone. The same model can behave differently depending on how it is routed.

Serving provider	What happens to `cache_control`
Anthropic, Google, Meta, Mistral and other routed models	Forwarded upstream. Applied on models that support caching and dropped on those that don’t.
Amazon Bedrock	Converted to a native cache point — but only for Claude and Amazon Nova models. Dropped for all other Bedrock models.
Direct adapters (OpenAI, Together, DeepInfra, Qwen, Vertex)	Stripped before the request is sent. These APIs reject or ignore the field.

OpenAI’s direct API performs prompt caching automatically on long prompts — you do not mark it, and you cannot control it from here. Markers are removed on that route not because caching is unavailable, but because the provider manages it itself.

An unsupported marker is silently removed — it never errors. Your request succeeds and returns a normal completion, and nothing in the response indicates the marker was dropped. Never assume a marker took effect: confirm it against cached_tokens in the response usage block.

On Bedrock a request may carry at most 4 cache breakpoints. If you send more, MeshAPI keeps the last 4 — later breakpoints cover longer prefixes, so keeping them preserves the most caching value. You can see which provider served any request in your usage records.

Reading the result

Every response tells you exactly what the cache did, under usage.prompt_tokens_details:

"usage": {
  "prompt_tokens": 6809,
  "completion_tokens": 40,
  "total_tokens": 6849,
  "prompt_tokens_details": {
    "cached_tokens": 6797,          // read from the cache — charged at 10%
    "cache_write_tokens": 0,        // written to the cache — charged at 125%
    "cache_write_1h_tokens": 0,     // the part of cache_write_tokens written with a 1h TTL
    "audio_tokens": 0
  },
  "completion_tokens_details": { "reasoning_tokens": 0, "audio_tokens": 0,
    "accepted_prediction_tokens": 0, "rejected_prediction_tokens": 0 },
  "byok_used": false,
  "fallback_triggered": false
}

Three states, and you should be able to tell them apart:

What you see	What happened
`cache_write_tokens` > 0	Cold call. The prefix was just stored; you paid the premium.
`cached_tokens` > 0	Hit. That many tokens were billed at the reduced rate.
`cached_tokens` and `cache_write_tokens` both 0	Nothing was cached at all — no marker, or one that did not take effect.

prompt_tokens includes both numbers, so the tokens processed fresh at the normal rate are prompt_tokens - cached_tokens - cache_write_tokens. Every key above is present on every chat completion, whichever provider served it, and is 0 when unused — you can index them directly. Two limits on what a 0 means:

cache_write_1h_tokens is filled only where the serving provider reports the 1h split. A route that doesn’t report it reads 0 even after a 1h write.
A route that does not report cache writes at all returns cache_write_tokens: 0.

Which routes report which counts, and the usage shapes on /v1/responses and /v1/messages, are in the Usage Object reference. total_tokens is always prompt_tokens + completion_tokens.

Deprecated: the top-level usage.cache_creation_input_tokens that Claude responses carry duplicates prompt_tokens_details.cache_write_tokens. It still appears for now; read cache_write_tokens instead.

On a streaming request the usage block arrives on the final chunk. You do not need to ask for it — MeshAPI requests usage from the provider on every stream, so the cache breakdown is there either way.

The per-request Usage API (/v1/usage, /v1/usage/events) returns no cache fields: not cached_tokens, and not the marker that distinguishes a gateway hit. Aggregates are available from GET /v1/analytics/usage/summary with metric=cached_tokens and metric=cache_hit_rate (an API key reads only its own traffic). For a per-request breakdown, record prompt_tokens_details from each response yourself.

What it costs

Cached tokens are not billed at the normal input rate. Every Claude model on MeshAPI uses the same two multipliers:

Token type	Rate	On `anthropic/claude-sonnet-5` ($2.00 / 1M input)
Cache read	0.10× input	$0.20 / 1M
Cache write	1.25× input	$2.50 / 1M
Ordinary input	1×	$2.00 / 1M

Measured end to end on anthropic/claude-sonnet-5 with a 6,800-token system block, same prompt each time:

Call	Result	Cost at list	vs uncached
First, with marker	6,797 written	$0.01742	1.24×
Second, with marker	6,797 read	$0.00178	0.13×
Same prompt, no marker	nothing cached	$0.01402	1.00×

So a hit costs about an eighth of the uncached call, and the write premium is repaid inside the first hit — two calls sharing a prefix are already ~31% cheaper than two uncached ones. What decides whether it pays is reuse, not prompt share. A segment costs 1.25× once and 0.10× on every later read, against 1.00× each time uncached — so it is ahead the first time it is read (1.25 + 0.10 < 2.00), however small a fraction of the prompt it is. A 2,000-token prefix on a 50,000-token request still pays, provided it is reused.

The way to lose money is a write that is never read. Each unread write costs 25% more than not caching at all. That happens when the marked region changes between calls, when the gap between calls exceeds the TTL, or when a marker sits after the part that varies. If you cannot say which later request will reuse the prefix, do not mark it.

Holding a prefix for an hour

The default entry expires after ~5 minutes. If your calls are further apart than that, add a TTL and the prefix survives an hour instead:

"cache_control": { "type": "ephemeral", "ttl": "1h" }

The longer hold costs more to create — 2× input instead of 1.25× — so it only pays when it converts writes into reads. Rule of thumb: if the gap between calls sharing a prefix is reliably under 5 minutes, stay on the default; if it’s typically longer, the 1h write is cheaper than paying a 5m write every time.

	5m (default)	1h
Write	1.25× input	2× input
Read	0.10× input	0.10× input
On `claude-sonnet-5`	$2.50 / 1M write	$4.00 / 1M write

Writes made under a 1h TTL are reported separately as cache_write_1h_tokens in prompt_tokens_details, so you can tell the two apart on the bill.

On Bedrock the TTL applies to Claude models only, and it is all-or-nothing per request: MeshAPI reports the 1h rate only when every breakpoint in the request carries ttl: "1h". Mix 5m and 1h markers in one call and the whole request bills at the 5m rate. A model whose price row has no 1h rate configured also bills 1h writes at the 5m rate.

Multi-turn: move the breakpoint forward

The biggest win is a conversation, not a single call. Each turn resends everything before it, so with one marker pinning the stable head and a second rolling onto the newest message, every turn reads the whole previous turn and writes only the delta.

"messages": [
  { "role": "system", "content": [
      { "type": "text", "text": "<stable instructions>",
        "cache_control": { "type": "ephemeral" } }
  ]},
  { "role": "user",      "content": "first question" },
  { "role": "assistant", "content": "..." },
  { "role": "user",      "content": "newest question",
    "cache_control": { "type": "ephemeral" } }
]

Move the second marker to the newest message on every turn rather than adding one per turn: the limit is 4. MeshAPI trims to the last 4 for you on Bedrock and the direct Anthropic route; OpenRouter forwards markers verbatim, so a 5th there is an upstream 400; every other route strips them entirely (see the table above). Measured over four turns with the model’s real replies fed back, turns 2–4 each read the entire previous prefix and wrote a 45–78 token tail, holding at 0.14× the uncached cost while the conversation grew.

When a marker doesn’t take effect

A marker that does nothing is silent — the request succeeds and returns a normal completion. If cached_tokens and cache_write_tokens both come back 0, work down this list:

The cached block is too small

Providers refuse to store a prefix below a minimum, and it is model-dependent: 1,024 tokens on Sonnet and Opus, 2,048 on Haiku. Under it you get no cache entry and no error. MeshAPI does not validate this for you.

Your tool list changed

Tool definitions sit ahead of the system block in the cached prefix, so editing, adding or removing a tool invalidates everything after it. Measured: the same system block with one extra tool schema returned cached_tokens: 0 and re-wrote all 7,153 tokens. Keep tools byte-stable across the calls meant to share a prefix — including the order of the array.

More than five minutes passed

A default ephemeral entry lives about 5 minutes, refreshed each time it is read. A loop that idles longer between turns pays the write again and again and never collects the discount — see Holding a prefix for an hour.

You trimmed the conversation from the front

Dropping the oldest messages to stay under a limit changes the start of the prompt, which is exactly what the cache keys on — so nothing after it can be reused. Summarize into the system block instead of dropping the head — otherwise every trim turns that turn’s write into one that is never read.

The route doesn't honour markers

See Where markers are honoured — direct OpenAI, Together, DeepInfra, Qwen and Vertex strip the field, and Bedrock applies it only to Claude and Nova. Check which provider actually served the request in your usage records.

A request that fails upstream does not warm the cache, so a rejected call costs you nothing in cache writes — the next valid call is a normal cold write.

Choosing between them

Repeating identical requests (dashboards, evals, retries, deterministic pipelines) → the gateway cache does this for free, automatically. Just keep temperature at 0.
Varying questions over a large fixed context (document Q&A, long system prompts, few-shot, agent loops) → the gateway cache never hits, because the messages differ every time. Use provider prompt caching: the prefix is reused on every question, which is exactly when it pays.
Prompts with nothing stable in them (each call unique, or calls further apart than the TTL) → neither. A marker here is written and never read, which costs 25% more than not caching.
Both are valid together: a repeated request hits the gateway cache first and never reaches the provider at all.

Memory — why memory-attached requests bypass the response cache
Usage & Monitoring API — measuring hit rate and savings
Usage Object — every usage field, and which cache counts each provider reports
Pricing — how cached tokens are billed

Introduction

Accounts & Access

Core Inference

Retrieval & Search

Multimodal

Operations

Mesh Code

Mesh CLI

Platform

Reference

Gateway response cache

What counts as “identical”

When caching applies

Detecting a cache hit

Turning it off

Cache isolation

Freshness

Provider prompt caching

Marking a prefix

Where markers are honoured

Reading the result

What it costs

Holding a prefix for an hour

Multi-turn: move the breakpoint forward

When a marker doesn’t take effect

Choosing between them

​Gateway response cache

​What counts as “identical”

​When caching applies

​Detecting a cache hit

​Turning it off

​Cache isolation

​Freshness

​Provider prompt caching

​Marking a prefix

​Where markers are honoured

​Reading the result

​What it costs

​Holding a prefix for an hour

​Multi-turn: move the breakpoint forward

​When a marker doesn’t take effect

​Choosing between them

​Related

Gateway response cache

What counts as “identical”

When caching applies

Detecting a cache hit

Turning it off

Cache isolation

Freshness

Provider prompt caching

Marking a prefix

Where markers are honoured

Reading the result

What it costs

Holding a prefix for an hour

Multi-turn: move the breakpoint forward

When a marker doesn’t take effect

Choosing between them

Related