Skip to main content
Median time added
10-11ms
One request at a time, both tiers
Median at 80 in flight
10ms
Slower of the two tiers
Share of a request
<5%
The model takes 93-99%
Requests completed
100%
5,380 sent, 0 rejected, 0 retried

From a November 2026 load test on the public endpoint: two model tiers, five load levels. Read the full report.

Every request you send to Mesh passes through the gateway on its way to the model. We call the time it spends there gateway overhead. The model’s own time depends on the model, the region and your prompt, so it isn’t part of this number.

How it holds up under load

Each row is a load level. The filled dot is the median request and the ring is the slowest 1% (p99). A short line means the gateway stayed steady. Hover a line for the exact values.
The median stays flat9-11 ms from 1 to 80 requests in flight, on both tiers. In October 2026 it was 24-32 ms.
Load shows up in the tailAt 80 in flight, the slowest 1% take up to 40 ms.
Bursts cost the most100 requests in the same instant push the median to 17-21 ms and the slowest 1% to 144 ms.
Nothing was droppedAll 5,380 requests finished. No rate-limit rejections and no retries.
Reading percentiles. p50 is the median request. p90 is slower than 90% of requests, p99 slower than 99%. At the single and steady levels, p99 rests on 50 and 40 requests, so one slow request moves it.

Where a request’s time goes

Here’s the median fast-tier request at each load level, split three ways. Almost all of it is the model.
On the frontier tier, which writes longer answers, the gateway is 0.6-1.1% of a request.

Streaming

The gateway’s overhead comes before the first token. After that, chunks pass straight through, and usage is recorded once the stream ends. In the test, the overhead was 9-11 ms at the median, and 17 ms in the spike.

Calls in a row

An agent that calls the model 10 times in sequence spends about 0.1 s in the gateway. Run independent calls in parallel, or let the response cache answer repeats.

What happens in those 10 ms

The gateway runs up to six steps on each request. A cache hit is answered at step 3 and skips the rest.
1Check your keyMatch your API key to its organization and confirm it’s active.Authentication
2Apply limitsCheck rate and spend limits for the key, the member and the organization.Rate limits
3Check the cacheAnswer an identical earlier request without calling the model.Caching
4RoutePick a provider and region, or a model when you ask for auto routing.Auto routing
5Retry or fall backHandle a provider error before it reaches you.Retry and fallback
6Record usageWrite tokens, cost and timings to your usage records.Usage object

Check it on your own requests

Every non-streamed response the model answers carries an X-Provider-Latency-Ms header: the model’s time for that request, in milliseconds. Take your total time, subtract the header, and what’s left is Mesh plus your network.
total_s is in seconds, so multiply it by 1,000 first. Run it a few times and compare medians, since the first request also sets up the connection.
Keep the X-Mesh-Cache: no-store header. Without it, repeat runs are answered from the response cache, which has no model time to subtract.

What changes your number

  • Distance to Mesh. The test client sat in one cloud region. A client farther away adds its own network time.
  • Bursts. Many requests in the same instant raise the tail, as the spike shows.
  • Gateway uptime. The test ran on freshly restarted gateway pods. On pods that had run for up to 22 hours, the median was the same but the slowest 1% reached 297 ms.
  • Features. Memory, RAG and prompt templates add work in the gateway. The test used plain chat completions.

How we tested

EndpointThe public endpoint, live models, response cache off
ClientOne machine in one cloud region
Requests5,380 in total, 2,690 per model tier
Model tiersFast tier non-streamed, frontier tier streamed
RequestChat completion, one-sentence prompt, 200 max tokens

Gateway overhead under load, November 2026

Every percentile for both tiers, the full setup, and how to run it yourself.

Gateway overhead under load, October 2026

The same test, run a month earlier.