Skip to main content
Median, up to 80 in flight
9-11ms
Both tiers, every level but the spike
Worst p99
144ms
Spike, frontier tier
Share of a request
1-5%
Median request, both tiers
Completed
5,380/ 5,380
0 rejected, 0 retried
In November 2026, we sent 5,380 requests through the public endpoint api.meshapi.ai and timed how long each one spent in the Mesh gateway, not counting the model. The load levels and the request are the same as in the October 2026 report. The fast tier used the same model, and the frontier tier used a different model of the same class. The overview explains what the gateway does in that time.

Results by load level

The filled dot is the median, the small dot p90, and the ring p99. The spike row is the worst case.
The median is a third of October’sFrom 1 to 80 requests in flight, it stays between 9 and 11 ms on both tiers. In October it was 24-32 ms.
The spike is the worst case100 requests with no ramp has a median of 17-21 ms and a p99 of 49-144 ms. In October those were 52-85 ms and 162-170 ms.
The tail stays short under loadAt 20 and 80 in flight, p99 is 22-40 ms on both tiers. In October it was 49-124 ms.
Long-running pods have a slower tailThe same test earlier the same day, on gateway pods that had run for up to 22 hours, had the same medians but p99 values up to 297 ms.

Every percentile

All values are in milliseconds. Each percentile is worked out separately for each level.
Non-streamed requests.
The single and steady levels have 50 and 40 requests, so their p99 is close to the one slowest request. Treat those two as a rough guide. Sustained and stress, at 500 and 2,000 requests, give a stable p99.

Where the time goes

The median fast-tier request at each level, split into the gateway, the model provider and the network between the client and Mesh.
The frontier tier writes longer answers, so the same gateway time is a smaller slice: 0.6-1.1% of the median request.

How busy the client kept each connection

With a fixed number of requests in flight, a client can finish at most (requests in flight / average response time) requests per second. The chart shows the measured rate as a share of that. Near 100% means the client kept every connection busy for the whole run, with little idle time at the start or the end.
Every request finished at every level. The stress level ran under a 150-second cap, and all 2,000 requests per tier completed inside it.

How we tested

DateNovember 2026
EndpointThe public endpoint, api.meshapi.ai
ClientOne load client in one cloud region
GatewayRestarted just before the test
ModelsLive provider models, not a mock, each served from the provider region nearest the client
Single50 requests, one at a time, after one untimed warm-up
Steady40 requests, 20 in flight
Sustained500 requests, 20 in flight
Spike100 requests sent in the same instant
Stress2,000 requests, 80 in flight, 150-second cap
Per tier2,690 requests
RequestChat completions, a one-sentence prompt, max_tokens 200
Fast tierNon-streamed, temperature 0.7
Frontier tierStreamed, model’s default temperature
CacheResponse cache off for every request, so every request reached the model
RetriesNone
TimingThe gateway splits each request into its own time and the provider call. The client measures the total. Network is the total minus both.

Run it yourself

1

Send the same request

Every request in the test used this body, with the tier’s model and stream setting. The fast tier also set temperature to 0.7.
2

Split out the gateway's time

Subtract the X-Provider-Latency-Ms response header from your total, as shown in Check it on your own requests.
3

Collect enough requests

Run each level for a few hundred requests before you compare p99 values.

What this test doesn’t cover

  • Other regions. One client in one region. A client elsewhere adds network time to each request, but not to gateway overhead.
  • Heavier requests. Plain chat completions only. Memory, RAG and prompt templates add gateway work.
  • Other request shapes. A longer prompt or answer changes the model’s time, and so the gateway’s share.