> ## Documentation Index
> Fetch the complete documentation index at: https://developers.meshapi.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Gateway overhead under load

> Tested in November 2026. 5,380 requests at five load levels, from one at a time to 100 at once, on a fast and a frontier model tier.

Median, up to 80 in flight9-11msBoth tiers, every level but the spike

Worst p99144msSpike, frontier tier

Share of a request1-5%Median request, both tiers

Completed5,380/ 5,3800 rejected, 0 retried

In November 2026, we sent 5,380 requests through the public endpoint `api.meshapi.ai` and timed how long each one spent in the Mesh gateway, not counting the model. The load levels and the request are the same as in the [October 2026 report](https://developers.meshapi.ai/performance/reports/gateway-execution-under-load.md). The fast tier used the same model, and the frontier tier used a different model of the same class. The [overview](https://developers.meshapi.ai/performance/overview.md) explains what the gateway does in that time.

## Results by load level

The filled dot is the median, the small dot p90, and the ring p99. The spike row is the worst case.

**The median is a third of October's**From 1 to 80 requests in flight, it stays between 9 and 11 ms on both tiers. In October it was 24-32 ms.**The spike is the worst case**100 requests with no ramp has a median of 17-21 ms and a p99 of 49-144 ms. In October those were 52-85 ms and 162-170 ms.**The tail stays short under load**At 20 and 80 in flight, p99 is 22-40 ms on both tiers. In October it was 49-124 ms.**Long-running pods have a slower tail**The same test earlier the same day, on gateway pods that had run for up to 22 hours, had the same medians but p99 values up to 297 ms.

## Every percentile

All values are in milliseconds. Each percentile is worked out separately for each level.

> [tab] Fast model tier
>
> Non-streamed requests.
>
> | Level | p50 | p90 | p99 |
> | - | - | - | - |
> | Single, 1 in flight | 10 | 13 | 16 |
> | Steady, 20 in flight | 11 | 23 | 26 |
> | Sustained, 20 in flight | 9 | 12 | 25 |
> | **Spike, 100 at once** | **21** | **35** | **49** |
> | Stress, 80 in flight | 10 | 16 | 37 |

> [tab] Frontier model tier
>
> Streamed requests, so the gateway's time is the time before the first token.
>
> | Level | p50 | p90 | p99 |
> | - | - | - | - |
> | Single, 1 in flight | 11 | 18 | 27 |
> | Steady, 20 in flight | 11 | 14 | 17 |
> | Sustained, 20 in flight | 9 | 11 | 22 |
> | **Spike, 100 at once** | **17** | **37** | **144** |
> | Stress, 80 in flight | 9 | 13 | 40 |

> [!NOTE]
>
> The single and steady levels have 50 and 40 requests, so their p99 is close to the one slowest request. Treat those two as a rough guide. Sustained and stress, at 500 and 2,000 requests, give a stable p99.

## Where the time goes

The median fast-tier request at each level, split into the gateway, the model provider and the network between the client and Mesh.

The frontier tier writes longer answers, so the same gateway time is a smaller slice: 0.6-1.1% of the median request.

## How busy the client kept each connection

With a fixed number of requests in flight, a client can finish at most (requests in flight / average response time) requests per second. The chart shows the measured rate as a share of that. Near 100% means the client kept every connection busy for the whole run, with little idle time at the start or the end.

Every request finished at every level. The stress level ran under a 150-second cap, and all 2,000 requests per tier completed inside it.

## How we tested

DateNovember 2026EndpointThe public endpoint, api.meshapi.aiClientOne load client in one cloud regionGatewayRestarted just before the testModelsLive provider models, not a mock, each served from the provider region nearest the clientSingle50 requests, one at a time, after one untimed warm-upSteady40 requests, 20 in flightSustained500 requests, 20 in flightSpike100 requests sent in the same instantStress2,000 requests, 80 in flight, 150-second capPer tier2,690 requestsRequestChat completions, a one-sentence prompt, max\_tokens 200Fast tierNon-streamed, temperature 0.7Frontier tierStreamed, model's default temperatureCacheResponse cache off for every request, so every request reached the modelRetriesNoneTimingThe gateway splits each request into its own time and the provider call. The client measures the total. Network is the total minus both.

## Run it yourself

**1. Send the same request**

Every request in the test used this body, with the tier's model and `stream` setting. The fast tier also set `temperature` to 0.7.

```json theme={null}
{
  "model": "<model>",
  "messages": [
    { "role": "user", "content": "In one sentence, what is a rate limit?" }
  ],
  "max_tokens": 200,
  "cache": false
}
```

**2. Split out the gateway's time**

Subtract the `X-Provider-Latency-Ms` response header from your total, as shown in [Check it on your own requests](https://developers.meshapi.ai/performance/overview.md#check-it-on-your-own-requests).

**3. Collect enough requests**

Run each level for a few hundred requests before you compare p99 values.

## What this test doesn't cover

* **Other regions.** One client in one region. A client elsewhere adds network time to each request, but not to gateway overhead.
* **Heavier requests.** Plain chat completions only. [Memory](https://developers.meshapi.ai/docs/capabilities/memory.md), [RAG](https://developers.meshapi.ai/docs/capabilities/rag.md) and [prompt templates](https://developers.meshapi.ai/docs/capabilities/prompt-templates.md) add gateway work.
* **Other request shapes.** A longer prompt or answer changes the model's time, and so the gateway's share.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.