Retry & Fallback

Upstream providers occasionally fail for reasons that have nothing to do with your request — a momentary 503, an overloaded region, a brief rate-limit spike. Mesh API absorbs these transient failures for you: instead of returning the first error it sees, the gateway automatically retries and, if needed, reroutes your request before responding. This is built into the platform and happens server-side. You don’t enable it, configure it, or change your code — you send a normal request and Mesh API does the resilience work behind the scenes.

Two independent layers

Alongside the gateway’s built-in behaviour, you can add resilience in the SDK. The two operate at different scopes and are safe to combine:

	Gateway routing policy	SDK client-side
Where it runs	Inside the gateway, before the response returns	In your process, around each request
Retry	Same provider/target	Same request to the gateway
Fallback	Same model, different provider	Different model (your chain)
Configured via	Dashboard, per key	Client config in code
Applies to	Non-streaming chat completions	Non-streaming chat completions

A typical setup: let the gateway absorb provider-level blips for a model transparently, with no code change, and use the SDK fallback chain to switch to an entirely different model if the primary is degraded everywhere. Neither layer retries streaming responses. See SDK Resilience.

What happens when a request fails

When an upstream returns an error, the gateway responds in escalating steps. It only moves to the next step if the previous one couldn’t recover:

Retry the same provider

The gateway waits briefly, then re-sends the request to the same provider. Transient blips often clear on the second or third try.

Fall back to the same model on a different provider

If the provider keeps failing, the gateway re-issues the request for the exact same model through a different upstream that serves it. You still get the model you asked for.

Fall back to a different model

If the requested model can’t be served anywhere right now, the gateway can route to a comparable alternate model so the request still succeeds.

All of this runs inside a single request. From your side, you make one call and receive one response — the retries and reroutes are invisible unless you look at the response headers or your dashboard logs.

Which errors trigger retry and fallback

Only transient failures — errors that could plausibly succeed on a second attempt — trigger retries and fallback. By default these are the HTTP status codes:

Status	Condition
`408`	Request Timeout
`429`	Too Many Requests (rate limited)
`500`	Internal Server Error
`502`	Bad Gateway
`503`	Service Unavailable
`504`	Gateway Timeout

The same set governs both same-provider retries and cross-provider fallback: a provider must return one of these (or time out with no response at all) for the gateway to try again.

Errors that are never retried

Some failures mean something is wrong with the request itself — trying again would only produce the same error. These are terminal: the gateway returns them immediately, with no retry and no fallback.

Category	Examples
Authentication / authorization	invalid or missing key, insufficient permissions (`401`, `403`)
Invalid input	malformed request, unsupported parameters, prompt over the model’s context limit (`400`)
Billing	insufficient credits or a billing block

A 429 (rate limit) is treated as transient and is retried, because it typically clears on its own or resolves once the request is routed to another provider.

How retries are paced

Retries against the same provider use exponential backoff with jitter — the gateway waits a short, randomized delay that grows with each attempt, so a provider recovering from a spike isn’t immediately hammered again. A provider is retried only a small number of times before the gateway moves on to fallback. The whole sequence runs inside an overall attempt and time budget. Retries and fallbacks stop once that budget is reached, so a struggling upstream can never make your request hang indefinitely — you get a result (or a final error) within a bounded window.

Per-key routing policy

Beyond the platform defaults, a key can carry an explicit routing policy. It is per-key opt-in — a key has no policy until you set one from the Dashboard (API Keys → edit a key → Resilience). Once set, the gateway applies it to that key’s non-streaming chat completions with no client changes. The policy is a JSON object with three blocks — retry, fallback, and budget:

{
  "retry": {
    "max_retries": 3,
    "retry_on_status": [429, 502, 503, 504],
    "base_ms": 500,
    "max_ms": 30000,
    "respect_retry_after": true,
    "per_attempt_timeout_ms": 60000
  },
  "fallback": {
    "enabled": true,
    "cross_provider_same_model": true,
    "models": ["anthropic/claude-sonnet-5"],
    "backoff_ms": 200,
    "on_status": [502, 503, 504]
  },
  "budget": {
    "max_attempts": 6,
    "max_total_latency_ms": 180000
  }
}

retry — how the gateway retries the same target before giving up or falling back.
fallback — when cross_provider_same_model is true, the gateway re-issues the request for the same model against a different provider. models optionally constrains the candidate set.
budget — a hard stop across all attempts, so retries and fallbacks can never run unbounded.

Ceilings

Values above these limits are clamped when you save a policy:

Field	Ceiling
`retry.max_retries`	≤ 6
`budget.max_attempts`	≤ 6
`budget.max_total_latency_ms`	≤ 180000 (180 s)

What each parameter controls

Decision	Controlled by
Is the policy active at all?	The key having a `routing_policy` set — no policy, no change in behaviour.
How many targets are in the plan?	`fallback.enabled` + `fallback.cross_provider_same_model` add the other providers serving the same model; `fallback.models` optionally constrains the candidates; the plan is capped at `budget.max_attempts` targets.
Is this target temporarily out of the pool?	The global circuit breaker (platform-managed, not per-key) — a skipped target falls through to the next one.
How long may a single attempt run?	`retry.per_attempt_timeout_ms`. An attempt exceeding it counts as a transient timeout. Unset means no per-attempt bound.
Never-retry safety rule	Not configurable. Auth, validation, and billing errors are terminal on any policy.
Retry the same target?	All three must hold: the status is in `retry.retry_on_status`, attempts so far are below `retry.max_retries` + 1 (and below `budget.max_attempts`), and the next attempt would still finish inside `budget.max_total_latency_ms`.
How long is the pause between retries?	Exponential backoff from `retry.base_ms`, doubling per attempt, capped at `retry.max_ms`, with jitter. With `retry.respect_retry_after`, a provider’s `Retry-After` hint overrides the computed delay.
Move to another provider?	`fallback.enabled` must be true and the error’s status must be in `fallback.on_status`. The same attempt budget and deadline keep applying.
Pause before the next provider	`fallback.backoff_ms` (`0` advances immediately).

Worked example

Policy: retry.max_retries = 1, fallback.enabled and cross_provider_same_model both true, budget.max_attempts = 4. The primary provider is having an outage:

Attempt 1 — primary provider returns 503. Transient, in retry_on_status, budget remains → retry.
Attempt 2 — after base_ms backoff, primary again returns 503. max_retries (1) is exhausted for this target → check fallback: eligible.
Attempt 3 — second provider, same model → 200 OK.

The response arrives with X-Mesh-Routing-Attempts: 3 and X-Mesh-Routing-Fallback: true, and the dashboard log row shows a Provider fallback badge and Retried ×2. Had attempt 1 failed with a 401 instead (terminal), it would have failed immediately — attempts 2 and 3 never happen, whatever the policy says.

Failing providers are taken out of rotation

Alongside per-request retries, Mesh API runs a platform-wide circuit breaker. When a specific model-and-provider combination starts failing repeatedly, it is temporarily pulled from the routing pool for everyone, then re-checked automatically after a short cooldown. This keeps requests from being routed into an upstream that is already known to be down, so fallback skips straight to a healthy provider. This is a global safety mechanism managed by the platform — it isn’t tied to any individual request or key.

A note on model fallback and billing

When the gateway falls back to the same model on a different provider, the model that answers is the one you requested, and billing is unchanged. When it falls back to a different model, you are billed for the model that actually served the request. In both cases the response tells you what happened so there are no surprises.

Streaming requests are protected only before the first token is sent. Once a stream has started emitting output, an error mid-stream is returned as-is — a partially streamed response can’t be retried or rerouted without duplicating output.

Seeing what happened

For non-streaming chat completions, the gateway reports what it did through response headers:

Header	Meaning
`X-Mesh-Routing-Attempts`	Total attempts made (the initial try plus any retries and fallback hops).
`X-Mesh-Routing-Fallback`	`true` if the request was ultimately served by a fallback rather than the primary provider.

Which specific upstream served the request is internal and isn’t reported — the headers tell you that a retry or fallback happened and how many attempts it took. Your Mesh API dashboard logs show the same information per request, including for streamed calls that don’t carry these headers.

Streaming responses do not carry these headers. Response headers are flushed before the SSE body starts, so any retries happen inside the stream generator after the headers have already been sent. For per-request routing detail on streamed calls, use the dashboard Logs — the log row shows the attempt count and a Provider fallback badge.

Not to be confused with `X-BYOK-Fallback-Triggered`

The X-Mesh-Routing-* headers describe routing fallback. That is a different mechanism from BYOK credential fallback:

Header	Mechanism	What “fallback” means
`X-Mesh-Routing-Fallback`	Resilient routing	The request was re-issued to the same model on a different provider.
`X-BYOK-Fallback-Triggered`	BYOK	Your own provider key failed, so the request fell back to Mesh API’s shared platform credentials.

Both can appear on the same response — a request can fall back from your BYOK credentials and be routed across providers — but they answer different questions. In the dashboard Logs they render as distinct BYOK fallback and Provider fallback badges.

Seeing it from the SDK

The Mesh API SDKs parse these headers into a structured gateway-routing event, so you can log exactly what the gateway did alongside your own client-side retries and fallbacks. See SDK Resilience.

The specific status codes, retry counts, timing, and fallback behaviour described here are the platform’s current defaults and may change in the future as we tune reliability. Treat retry and fallback as a best-effort resilience layer rather than a fixed contract, and always handle a final error response in your own code.

Introduction

Accounts & Access

Core Inference

Retrieval & Search

Multimodal

Operations

Mesh CLI

Platform

Reference

Two independent layers

What happens when a request fails

Which errors trigger retry and fallback

Errors that are never retried

How retries are paced

Per-key routing policy

Ceilings

What each parameter controls

Worked example

Failing providers are taken out of rotation

A note on model fallback and billing

Seeing what happened

Not to be confused with `X-BYOK-Fallback-Triggered`

Seeing it from the SDK

​Two independent layers

​What happens when a request fails

​Which errors trigger retry and fallback

​Errors that are never retried

​How retries are paced

​Per-key routing policy

​Ceilings

​What each parameter controls

​Worked example

​Failing providers are taken out of rotation

​A note on model fallback and billing

​Seeing what happened

​Not to be confused with X-BYOK-Fallback-Triggered

​Seeing it from the SDK

Two independent layers

What happens when a request fails

Which errors trigger retry and fallback

Errors that are never retried

How retries are paced

Per-key routing policy

Ceilings

What each parameter controls

Worked example

Failing providers are taken out of rotation

A note on model fallback and billing

Seeing what happened

Not to be confused with `X-BYOK-Fallback-Triggered`

Seeing it from the SDK