> ## Documentation Index
> Fetch the complete documentation index at: https://developers.meshapi.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Gateway overhead under load

> Tested in October 2026. 5,380 requests at five load levels, from one at a time to 100 at once, on a fast and a frontier model tier.

Median, up to 80 in flight24-32msBoth tiers, every level but the spike

Worst p99170msSpike, frontier tier

Share of a request1-8%Median request, both tiers

Completed5,380/ 5,3800 rejected, 0 retried

> [!NOTE]
>
> A newer run of the same test is in the [November 2026 report](https://developers.meshapi.ai/performance/reports/gateway-overhead-under-load.md).

In October 2026, we sent 5,380 requests through the public endpoint `api.meshapi.ai` and timed how long each one spent in the Mesh gateway, not counting the model. The [overview](https://developers.meshapi.ai/performance/overview.md) explains what the gateway does in that time.

## Results by load level

The filled dot is the median, the small dot p90, and the ring p99. The spike row is the worst case.

**The median barely moves**From 1 to 80 requests in flight, it stays between 24 and 32 ms on both tiers.**Load shows up in the tail first**Fast-tier p99 goes from 41 ms one at a time to 124 ms at 80 in flight.**The spike is the worst case**100 requests with no ramp is the highest reading at every percentile, on both tiers.**Outside the spike, the tiers match**The two tiers' medians are within 4 ms of each other at every other level. Only the gateway's share of the request changes.

## Every percentile

All values are in milliseconds. Each percentile is worked out separately for each level.

> [tab] Fast model tier
>
> Non-streamed requests.
>
> | Level | p50 | p90 | p99 |
> | - | - | - | - |
> | Single, 1 in flight | 27 | 32 | 41 |
> | Steady, 20 in flight | 32 | 59 | 65 |
> | Sustained, 20 in flight | 27 | 38 | 86 |
> | **Spike, 100 at once** | **52** | **113** | **162** |
> | Stress, 80 in flight | 29 | 52 | 124 |

> [tab] Frontier model tier
>
> Streamed requests, so the gateway's time is the time before the first token.
>
> | Level | p50 | p90 | p99 |
> | - | - | - | - |
> | Single, 1 in flight | 28 | 40 | 70 |
> | Steady, 20 in flight | 30 | 46 | 63 |
> | Sustained, 20 in flight | 24 | 33 | 49 |
> | **Spike, 100 at once** | **85** | **140** | **170** |
> | Stress, 80 in flight | 25 | 36 | 97 |

> [!NOTE]
>
> The single and steady levels have 50 and 40 requests, so their p99 is close to the one slowest request. Treat those two as a rough guide. Sustained and stress, at 500 and 2,000 requests, give a stable p99.

## Where the time goes

The median fast-tier request at each level, split into the gateway, the model provider and the network between the client and Mesh.

The frontier tier writes longer answers, so the same gateway time is a smaller slice: 1.3-4.4% of the median request. The network share is largest at the steady and spike levels.

## How busy the client kept each connection

With a fixed number of requests in flight, a client can finish at most (requests in flight / average response time) requests per second. The chart shows the measured rate as a share of that. Near 100% means the client kept every connection busy for the whole run, with little idle time at the start or the end.

Every request finished at every level. The stress level ran under a 150-second cap, and all 2,000 requests per tier completed inside it.

## How we tested

DateOctober 2026EndpointThe public endpoint, api.meshapi.aiClientOne load client in one cloud regionModelsLive provider models, not a mock, each served from the provider region nearest the clientSingle50 requests, one at a time, after one untimed warm-upSteady40 requests, 20 in flightSustained500 requests, 20 in flightSpike100 requests sent in the same instantStress2,000 requests, 80 in flight, 150-second capPer tier2,690 requestsRequestChat completions, a one-sentence prompt, max\_tokens 200Fast tierNon-streamed, temperature 0.7Frontier tierStreamed, model's default temperatureCacheResponse cache off for every request, so every request reached the modelRetriesNoneTimingThe gateway splits each request into its own time and the provider call. The client measures the total. Network is the total minus both.

## Run it yourself

**1. Send the same request**

Every request in the test used this body, with the tier's model and `stream` setting. The fast tier also set `temperature` to 0.7.

```json theme={null}
{
  "model": "<model>",
  "messages": [
    { "role": "user", "content": "In one sentence, what is a rate limit?" }
  ],
  "max_tokens": 200,
  "cache": false
}
```

**2. Split out the gateway's time**

Subtract the `X-Provider-Latency-Ms` response header from your total, as shown in [Check it on your own requests](https://developers.meshapi.ai/performance/overview.md#check-it-on-your-own-requests).

**3. Collect enough requests**

Run each level for a few hundred requests before you compare p99 values.

## What this test doesn't cover

* **Other regions.** One client in one region. A client elsewhere adds network time to each request, but not to gateway overhead.
* **Heavier requests.** Plain chat completions only. [Memory](https://developers.meshapi.ai/docs/capabilities/memory.md), [RAG](https://developers.meshapi.ai/docs/capabilities/rag.md) and [prompt templates](https://developers.meshapi.ai/docs/capabilities/prompt-templates.md) add gateway work.
* **Other request shapes.** A longer prompt or answer changes the model's time, and so the gateway's share.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.