> ## Documentation Index
> Fetch the complete documentation index at: https://developers.meshapi.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Mesh adds about 10 ms to a request

> That's the time a request spends in the Mesh gateway, not counting the model. It stays near 10 ms up to 80 requests in flight.

Median time added10-11msOne request at a time, both tiers

Median at 80 in flight10msSlower of the two tiers

Share of a request\<5%The model takes 93-99%

Requests completed100%5,380 sent, 0 rejected, 0 retried

From a November 2026 load test on the public endpoint: two model tiers, five load levels. [Read the full report](https://developers.meshapi.ai/performance/reports/gateway-overhead-under-load.md).

Every request you send to Mesh passes through the gateway on its way to the model. We call the time it spends there **gateway overhead**. The model's own time depends on the model, the region and your prompt, so it isn't part of this number.

## How it holds up under load

Each row is a load level. The filled dot is the median request and the ring is the slowest 1% (p99). A short line means the gateway stayed steady. Hover a line for the exact values.

**The median stays flat**9-11 ms from 1 to 80 requests in flight, on both tiers. In October 2026 it was 24-32 ms.**Load shows up in the tail**At 80 in flight, the slowest 1% take up to 40 ms.**Bursts cost the most**100 requests in the same instant push the median to 17-21 ms and the slowest 1% to 144 ms.**Nothing was dropped**All 5,380 requests finished. No rate-limit rejections and no retries.

| Level | What it sends | Looks like |
| - | - | - |
| Single | 50 requests, one at a time | One person chatting |
| Steady | 40 requests, 20 in flight | A small team at once |
| Sustained | 500 requests, 20 in flight | A backend job working through a queue |
| Spike | 100 requests in the same instant | An agent fanning out many calls |
| Stress | 2,000 requests, 80 in flight | A busy production service |

> [!TIP]
>
> **Reading percentiles.** p50 is the median request. p90 is slower than 90% of requests, p99 slower than 99%. At the single and steady levels, p99 rests on 50 and 40 requests, so one slow request moves it.

## Where a request's time goes

Here's the median fast-tier request at each load level, split three ways. Almost all of it is the model.

On the frontier tier, which writes longer answers, the gateway is 0.6-1.1% of a request.

> [card] Streaming
>
> The gateway's overhead comes before the first token. After that, chunks pass straight through, and usage is recorded once the stream ends. In the test, the overhead was 9-11 ms at the median, and 17 ms in the spike.

> [card] Calls in a row
>
> An agent that calls the model 10 times in sequence spends about 0.1 s in the gateway. Run independent calls in parallel, or let the [response cache](https://developers.meshapi.ai/docs/capabilities/caching.md) answer repeats.

## What happens in those 10 ms

The gateway runs up to six steps on each request. A cache hit is answered at step 3 and skips the rest.

*1***Check your key**Match your API key to its organization and confirm it's active.[Authentication](https://developers.meshapi.ai/docs/getting-started/authentication.md)*2***Apply limits**Check rate and spend limits for the key, the member and the organization.[Rate limits](https://developers.meshapi.ai/docs/getting-started/rate-limits.md)*3***Check the cache**Answer an identical earlier request without calling the model.[Caching](https://developers.meshapi.ai/docs/capabilities/caching.md)*4***Route**Pick a provider and region, or a model when you ask for auto routing.[Auto routing](https://developers.meshapi.ai/docs/capabilities/auto-routing.md)*5***Retry or fall back**Handle a provider error before it reaches you.[Retry and fallback](https://developers.meshapi.ai/docs/platform/retry-and-fallback.md)*6***Record usage**Write tokens, cost and timings to your usage records.[Usage object](https://developers.meshapi.ai/docs/reference/usage-object.md)

## Check it on your own requests

Every non-streamed response the model answers carries an `X-Provider-Latency-Ms` header: the model's time for that request, in milliseconds. Take your total time, subtract the header, and what's left is Mesh plus your network.

```bash theme={null}
curl -s -o /dev/null -D - -w "total_s=%{time_total}\n" \
  https://api.meshapi.ai/v1/chat/completions \
  -H "Authorization: Bearer rsk_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -H "X-Mesh-Cache: no-store" \
  -d '{"model": "auto", "messages": [{"role": "user", "content": "In one sentence, what is a rate limit?"}]}' \
  | grep -iE "x-provider-latency-ms|total_s"
```

`total_s` is in seconds, so multiply it by 1,000 first. Run it a few times and compare medians, since the first request also sets up the connection.

> [!WARNING]
>
> Keep the `X-Mesh-Cache: no-store` header. Without it, repeat runs are answered from the [response cache](https://developers.meshapi.ai/docs/capabilities/caching.md), which has no model time to subtract.

## What changes your number

* **Distance to Mesh.** The test client sat in one cloud region. A client farther away adds its own network time.
* **Bursts.** Many requests in the same instant raise the tail, as the spike shows.
* **Gateway uptime.** The test ran on freshly restarted gateway pods. On pods that had run for up to 22 hours, the median was the same but the slowest 1% reached 297 ms.
* **Features.** [Memory](https://developers.meshapi.ai/docs/capabilities/memory.md), [RAG](https://developers.meshapi.ai/docs/capabilities/rag.md) and [prompt templates](https://developers.meshapi.ai/docs/capabilities/prompt-templates.md) add work in the gateway. The test used plain chat completions.

## How we tested

EndpointThe public endpoint, live models, response cache offClientOne machine in one cloud regionRequests5,380 in total, 2,690 per model tierModel tiersFast tier non-streamed, frontier tier streamedRequestChat completion, one-sentence prompt, 200 max tokens

> [card] [Gateway overhead under load, November 2026](https://developers.meshapi.ai/performance/reports/gateway-overhead-under-load.md)
>
> Every percentile for both tiers, the full setup, and how to run it yourself.

> [card] [Gateway overhead under load, October 2026](https://developers.meshapi.ai/performance/reports/gateway-execution-under-load.md)
>
> The same test, run a month earlier.

## Related

* [Caching](https://developers.meshapi.ai/docs/capabilities/caching.md): skip the model for repeated requests
* [Retry and fallback](https://developers.meshapi.ai/docs/platform/retry-and-fallback.md): what happens when a provider fails
* [Rate limits](https://developers.meshapi.ai/docs/getting-started/rate-limits.md): the limits checked on each request
* [Usage object](https://developers.meshapi.ai/docs/reference/usage-object.md): the timing and token fields on each response


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.