> ## Documentation Index
> Fetch the complete documentation index at: https://developers.meshapi.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Retry & Fallback
> How Mesh API automatically retries and reroutes requests when an upstream provider returns a transient error — which status codes trigger it, and how same-provider retries, cross-provider fallback, and model fallback fit together.
Upstream providers occasionally fail for reasons that have nothing to do with
your request — a momentary `503`, an overloaded region, a brief rate-limit
spike. Mesh API absorbs these transient failures for you: instead of returning
the first error it sees, the gateway automatically **retries** and, if needed,
**reroutes** your request before responding.
This is built into the platform and happens **server-side**. You don't enable
it, configure it, or change your code — you send a normal request and Mesh API
does the resilience work behind the scenes.
## Two independent layers
Alongside the gateway's built-in behaviour, you can add resilience in the SDK.
The two operate at different scopes and are safe to combine:
| | Gateway routing policy | SDK client-side |
| -------------- | ----------------------------------------------- | ------------------------------------ |
| Where it runs | Inside the gateway, before the response returns | In your process, around each request |
| Retry | Same provider/target | Same request to the gateway |
| Fallback | **Same model, different provider** | **Different model** (your chain) |
| Configured via | Dashboard, per key | Client config in code |
| Applies to | Non-streaming chat completions | Non-streaming chat completions |
A typical setup: let the **gateway** absorb provider-level blips for a model
transparently, with no code change, and use the **SDK fallback chain** to switch
to an entirely different model if the primary is degraded everywhere. Neither
layer retries streaming responses. See [SDK Resilience](/sdk/resilience).
## What happens when a request fails
When an upstream returns an error, the gateway responds in escalating steps.
It only moves to the next step if the previous one couldn't recover:
The gateway waits briefly, then re-sends the request to the same provider.
Transient blips often clear on the second or third try.
If the provider keeps failing, the gateway re-issues the request for the
**exact same model** through a different upstream that serves it. You still
get the model you asked for.
If the requested model can't be served anywhere right now, the gateway can
route to a comparable alternate model so the request still succeeds.
All of this runs inside a single request. From your side, you make one call and
receive one response — the retries and reroutes are invisible unless you look
at the [response headers](#seeing-what-happened) or your dashboard logs.
```mermaid theme={null}
flowchart TD
A[Request arrives] --> E[Send to primary provider]
E --> F{Result?}
F -- Success --> S[Return response ✓]
F -- "Terminal error
auth · invalid input · billing" --> X1[Return error immediately ✗]
F -- "Transient error
408 · 429 · 500 · 502 · 503 · 504" --> H{Retries left
for this provider?}
H -- yes --> I[Back off, then retry same provider]
I --> E
H -- no --> J{Same model available
on another provider?}
J -- yes --> K[Retry the same model
on a different provider]
K --> F
J -- no --> L{A fallback model
available?}
L -- yes --> M[Try a different model]
M --> F
L -- no --> X2[Return the last error ✗]
```
## Which errors trigger retry and fallback
Only **transient** failures — errors that could plausibly succeed on a second
attempt — trigger retries and fallback. By default these are the HTTP status
codes:
| Status | Condition |
| ------ | -------------------------------- |
| `408` | Request Timeout |
| `429` | Too Many Requests (rate limited) |
| `500` | Internal Server Error |
| `502` | Bad Gateway |
| `503` | Service Unavailable |
| `504` | Gateway Timeout |
The same set governs both same-provider retries and cross-provider fallback: a
provider must return one of these (or time out with no response at all) for the
gateway to try again.
### Errors that are never retried
Some failures mean something is wrong with the request itself — trying again
would only produce the same error. These are **terminal**: the gateway returns
them immediately, with no retry and no fallback.
| Category | Examples |
| ---------------------------------- | ---------------------------------------------------------------------------------------- |
| **Authentication / authorization** | invalid or missing key, insufficient permissions (`401`, `403`) |
| **Invalid input** | malformed request, unsupported parameters, prompt over the model's context limit (`400`) |
| **Billing** | insufficient credits or a billing block |
A `429` (rate limit) is treated as transient and *is* retried, because it
typically clears on its own or resolves once the request is routed to another
provider.
## How retries are paced
Retries against the same provider use **exponential backoff with jitter** — the
gateway waits a short, randomized delay that grows with each attempt, so a
provider recovering from a spike isn't immediately hammered again. A provider is
retried only a small number of times before the gateway moves on to fallback.
The whole sequence runs inside an overall **attempt and time budget**. Retries
and fallbacks stop once that budget is reached, so a struggling upstream can
never make your request hang indefinitely — you get a result (or a final error)
within a bounded window.
## Per-key routing policy
Beyond the platform defaults, a key can carry an explicit **routing policy**. It is
per-key opt-in — a key has no policy until you set one from the
[Dashboard](https://app.meshapi.ai) (**API Keys → edit a key → Resilience**). Once
set, the gateway applies it to that key's non-streaming chat completions with no
client changes.
The policy is a JSON object with three blocks — `retry`, `fallback`, and `budget`:
```json theme={null}
{
"retry": {
"max_retries": 3,
"retry_on_status": [429, 502, 503, 504],
"base_ms": 500,
"max_ms": 30000,
"respect_retry_after": true,
"per_attempt_timeout_ms": 60000
},
"fallback": {
"enabled": true,
"cross_provider_same_model": true,
"models": ["anthropic/claude-sonnet-5"],
"backoff_ms": 200,
"on_status": [502, 503, 504]
},
"budget": {
"max_attempts": 6,
"max_total_latency_ms": 180000
}
}
```
* **`retry`** — how the gateway retries the *same* target before giving up or falling back.
* **`fallback`** — when `cross_provider_same_model` is `true`, the gateway re-issues the request for the **same model against a different provider**. `models` optionally constrains the candidate set.
* **`budget`** — a hard stop across all attempts, so retries and fallbacks can never run unbounded.
### Ceilings
Values above these limits are clamped when you save a policy:
| Field | Ceiling |
| ----------------------------- | ---------------- |
| `retry.max_retries` | ≤ 6 |
| `budget.max_attempts` | ≤ 6 |
| `budget.max_total_latency_ms` | ≤ 180000 (180 s) |
### What each parameter controls
| Decision | Controlled by |
| ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Is the policy active at all? | The key having a `routing_policy` set — no policy, no change in behaviour. |
| How many targets are in the plan? | `fallback.enabled` + `fallback.cross_provider_same_model` add the other providers serving the same model; `fallback.models` optionally constrains the candidates; the plan is capped at `budget.max_attempts` targets. |
| Is this target temporarily out of the pool? | The **global** circuit breaker (platform-managed, not per-key) — a skipped target falls through to the next one. |
| How long may a single attempt run? | `retry.per_attempt_timeout_ms`. An attempt exceeding it counts as a transient timeout. Unset means no per-attempt bound. |
| Never-retry safety rule | Not configurable. Auth, validation, and billing errors are terminal on any policy. |
| Retry the same target? | All three must hold: the status is in `retry.retry_on_status`, attempts so far are below `retry.max_retries` + 1 (and below `budget.max_attempts`), and the next attempt would still finish inside `budget.max_total_latency_ms`. |
| How long is the pause between retries? | Exponential backoff from `retry.base_ms`, doubling per attempt, capped at `retry.max_ms`, with jitter. With `retry.respect_retry_after`, a provider's `Retry-After` hint overrides the computed delay. |
| Move to another provider? | `fallback.enabled` must be true and the error's status must be in `fallback.on_status`. The same attempt budget and deadline keep applying. |
| Pause before the next provider | `fallback.backoff_ms` (`0` advances immediately). |
### Worked example
Policy: `retry.max_retries = 1`, `fallback.enabled` and `cross_provider_same_model`
both `true`, `budget.max_attempts = 4`. The primary provider is having an outage:
1. **Attempt 1** — primary provider returns `503`. Transient, in `retry_on_status`, budget remains → retry.
2. **Attempt 2** — after `base_ms` backoff, primary again returns `503`. `max_retries` (1) is exhausted for this target → check fallback: eligible.
3. **Attempt 3** — second provider, same model → `200 OK`.
The response arrives with `X-Mesh-Routing-Attempts: 3` and
`X-Mesh-Routing-Fallback: true`, and the dashboard log row shows a **Provider
fallback** badge and *Retried ×2*. Had attempt 1 failed with a `401` instead
(terminal), it would have failed immediately — attempts 2 and 3 never happen,
whatever the policy says.
## Failing providers are taken out of rotation
Alongside per-request retries, Mesh API runs a platform-wide **circuit
breaker**. When a specific model-and-provider combination starts failing
repeatedly, it is temporarily pulled from the routing pool for everyone, then
re-checked automatically after a short cooldown. This keeps requests from being
routed into an upstream that is already known to be down, so fallback skips
straight to a healthy provider.
This is a global safety mechanism managed by the platform — it isn't tied to any
individual request or key.
## A note on model fallback and billing
When the gateway falls back to the **same model on a different provider**, the
model that answers is the one you requested, and billing is unchanged.
When it falls back to a **different model**, you are billed for the model that
actually served the request. In both cases the response tells you what happened
so there are no surprises.
**Streaming requests** are protected only *before the first token is sent*.
Once a stream has started emitting output, an error mid-stream is returned
as-is — a partially streamed response can't be retried or rerouted without
duplicating output.
## Seeing what happened
For **non-streaming** chat completions, the gateway reports what it did through
response headers:
| Header | Meaning |
| ------------------------- | ------------------------------------------------------------------------------------------- |
| `X-Mesh-Routing-Attempts` | Total attempts made (the initial try plus any retries and fallback hops). |
| `X-Mesh-Routing-Fallback` | `true` if the request was ultimately served by a fallback rather than the primary provider. |
Which specific upstream served the request is internal and isn't reported — the
headers tell you *that* a retry or fallback happened and how many attempts it
took. Your Mesh API dashboard logs show the same information per request,
including for streamed calls that don't carry these headers.
**Streaming responses do not carry these headers.** Response headers are
flushed before the SSE body starts, so any retries happen inside the stream
generator after the headers have already been sent. For per-request routing
detail on streamed calls, use the dashboard [Logs](https://app.meshapi.ai) —
the log row shows the attempt count and a **Provider fallback** badge.
### Not to be confused with `X-BYOK-Fallback-Triggered`
The `X-Mesh-Routing-*` headers describe **routing** fallback. That is a different
mechanism from [BYOK](/docs/capabilities/byok) **credential** fallback:
| Header | Mechanism | What "fallback" means |
| --------------------------- | ------------------------------- | ----------------------------------------------------------------------------------------------------- |
| `X-Mesh-Routing-Fallback` | Resilient routing | The request was re-issued to the same model on a **different provider**. |
| `X-BYOK-Fallback-Triggered` | [BYOK](/docs/capabilities/byok) | Your own provider key failed, so the request fell back to Mesh API's **shared platform credentials**. |
Both can appear on the same response — a request can fall back from your BYOK
credentials *and* be routed across providers — but they answer different
questions. In the dashboard Logs they render as distinct **BYOK fallback** and
**Provider fallback** badges.
### Seeing it from the SDK
The Mesh API SDKs parse these headers into a structured `gateway-routing` event,
so you can log exactly what the gateway did alongside your own client-side
retries and fallbacks. See [SDK Resilience](/sdk/resilience).
The specific status codes, retry counts, timing, and fallback behaviour
described here are the platform's current defaults and may change in the
future as we tune reliability. Treat retry and fallback as a best-effort
resilience layer rather than a fixed contract, and always handle a final
error response in your own code.