Skip to main content
Median, up to 80 in flight
24-32ms
Both tiers, every level but the spike
Worst p99
170ms
Spike, frontier tier
Share of a request
1-8%
Median request, both tiers
Completed
5,380/ 5,380
0 rejected, 0 retried
A newer run of the same test is in the November 2026 report.
In October 2026, we sent 5,380 requests through the public endpoint api.meshapi.ai and timed how long each one spent in the Mesh gateway, not counting the model. The overview explains what the gateway does in that time.

Results by load level

The filled dot is the median, the small dot p90, and the ring p99. The spike row is the worst case.
The median barely movesFrom 1 to 80 requests in flight, it stays between 24 and 32 ms on both tiers.
Load shows up in the tail firstFast-tier p99 goes from 41 ms one at a time to 124 ms at 80 in flight.
The spike is the worst case100 requests with no ramp is the highest reading at every percentile, on both tiers.
Outside the spike, the tiers matchThe two tiers’ medians are within 4 ms of each other at every other level. Only the gateway’s share of the request changes.

Every percentile

All values are in milliseconds. Each percentile is worked out separately for each level.
Non-streamed requests.
The single and steady levels have 50 and 40 requests, so their p99 is close to the one slowest request. Treat those two as a rough guide. Sustained and stress, at 500 and 2,000 requests, give a stable p99.

Where the time goes

The median fast-tier request at each level, split into the gateway, the model provider and the network between the client and Mesh.
The frontier tier writes longer answers, so the same gateway time is a smaller slice: 1.3-4.4% of the median request. The network share is largest at the steady and spike levels.

How busy the client kept each connection

With a fixed number of requests in flight, a client can finish at most (requests in flight / average response time) requests per second. The chart shows the measured rate as a share of that. Near 100% means the client kept every connection busy for the whole run, with little idle time at the start or the end.
Every request finished at every level. The stress level ran under a 150-second cap, and all 2,000 requests per tier completed inside it.

How we tested

DateOctober 2026
EndpointThe public endpoint, api.meshapi.ai
ClientOne load client in one cloud region
ModelsLive provider models, not a mock, each served from the provider region nearest the client
Single50 requests, one at a time, after one untimed warm-up
Steady40 requests, 20 in flight
Sustained500 requests, 20 in flight
Spike100 requests sent in the same instant
Stress2,000 requests, 80 in flight, 150-second cap
Per tier2,690 requests
RequestChat completions, a one-sentence prompt, max_tokens 200
Fast tierNon-streamed, temperature 0.7
Frontier tierStreamed, model’s default temperature
CacheResponse cache off for every request, so every request reached the model
RetriesNone
TimingThe gateway splits each request into its own time and the provider call. The client measures the total. Network is the total minus both.

Run it yourself

1

Send the same request

Every request in the test used this body, with the tier’s model and stream setting. The fast tier also set temperature to 0.7.
2

Split out the gateway's time

Subtract the X-Provider-Latency-Ms response header from your total, as shown in Check it on your own requests.
3

Collect enough requests

Run each level for a few hundred requests before you compare p99 values.

What this test doesn’t cover

  • Other regions. One client in one region. A client elsewhere adds network time to each request, but not to gateway overhead.
  • Heavier requests. Plain chat completions only. Memory, RAG and prompt templates add gateway work.
  • Other request shapes. A longer prompt or answer changes the model’s time, and so the gateway’s share.