Routing Algorithms

Auto Routing covers how to use model: "auto". This page covers how the pick is actually made, and the one knob you can turn.

Two layers

A routing decision goes through two layers, and the word “weight” means something different in each:

Which algorithm chain runs — traffic is split across named paths by traffic weights. Operator-configured; you observe the result, you don’t set it.
Which model that algorithm picks — the weighted algorithm scores every candidate with scoring weights over quality, cost, and latency. You can influence this per request, with weight_profile.

The algorithms

Each path is an ordered waterfall. Algorithms run in sequence and the first one that returns a model wins; an algorithm that declines (“abstains”) costs nothing and falls through to the next. Every algorithm has its own timeout, and none of them can fail your request — see Nothing here can fail a request.

Algorithm	How it picks	LLM call	Typical use
`heuristic`	String-gates trivial prompts (greetings, one-line questions) and routes straight to a configured fast-lane model	No	A fast lane in front of the others, with no classify round-trip
`weighted`	Classifies, then scores the candidate pool on quality, cost and latency and takes the argmax	Yes (classifier)	The main selector
`benchmark`	Classifies into a task category, then takes the top-ranked model for that category	Yes (classifier)	Quality-led picking; also the `weighted` fallback
`registry`	Shows the full candidate list to a classifier, which picks by reading model descriptions	Yes (classifier)	Last resort when the category ranking has nothing
`not_diamond`	Sends the prompt to Not Diamond’s trained router, which picks among the candidates it knows	Yes (external router)	An optional leg; abstains when fewer than two candidates are known to it

Which algorithms run, and in what order, is set by the active configuration. See Traffic weights.

heuristic is deliberately conservative. Any hint of task work, recency, code, links, more than six digits, more than one line or length over 140 characters makes it decline. So does a reference to an attachment the fast-lane model can’t read, or a prompt that is neither a question nor small talk. The request then takes the full path instead. Over-declining is harmless; a wrong fast-lane pick is not.

What the pool is

Every algorithm picks from the same candidate pool: the models the gateway currently serves for that surface, already filtered by your key’s model policy and the request’s capability requirements. An algorithm can only ever return a model you were entitled to call directly.

Traffic weights

The active configuration names one or more paths, each with a relative weight, and a path is chosen per request by weighted random.

[
  { "name": "weighted-first",  "weight": 90,
    "algorithms": [ { "name": "heuristic", "timeout_ms": 200 },
                    { "name": "weighted",  "timeout_ms": 5000 },
                    { "name": "benchmark", "timeout_ms": 5000 },
                    { "name": "registry",  "timeout_ms": 5000 } ] },
  { "name": "benchmark-first", "weight": 10,
    "algorithms": [ { "name": "heuristic", "timeout_ms": 200 },
                    { "name": "benchmark", "timeout_ms": 5000 },
                    { "name": "registry",  "timeout_ms": 5000 } ] }
]

Weights are relative positive integers, normalised internally — [90, 10] and [9, 1] are identical. The shape above is illustrative; the live split is an operating decision and changes without an API change.

Scoring weights

When the weighted algorithm runs, it scores every candidate in the pool:

score(m) = w_q · Q(m)  +  w_c · C(m)  +  w_l · L(m)

Highest score wins. The runners-up become the fallover chain, so if the winner fails the request moves down the same ranking rather than to an unrelated model.

The three signals

All three are normalised so higher is always better, which is why cost and latency are inverted — a cheap model scores high on C, a fast one scores high on L.

Signal	Source	Normalisation
Q — quality	The model’s position in the ranking for the classified task category. Candidates with a measured benchmark index (coding / math / general, chosen by category) are reordered within the slots their cohort already holds — best index first, ties broken on curated rank	`1 - pos/(N-1)` over the pool
C — cost	Blended list price per 1M tokens: `0.3 × prompt + 0.7 × completion`	Log-scaled, then min-max inverted
L — latency	Observed p95 for that model — time-to-first-token on streaming requests, full wall-clock otherwise	Min-max inverted

Only the order of a measured index matters, never its magnitude, and a candidate with no measured score keeps its curated position rather than dropping. Nothing falls out of the pool for want of a benchmark.

Missing signals degrade, they don’t error

A candidate missing a signal drops that term, and its remaining weights are renormalised to sum to 1. A candidate with neither cost nor latency data is scored on quality alone — so when the whole pool has no cost or latency data, weighted returns exactly what benchmark would have.

Weight profiles

The (w_q, w_c, w_l) vector is named. Four profiles ship:

Profile	w_q (quality)	w_c (cost)	w_l (latency)	Picks
`quality_first`	0.75	0.15	0.10	The strongest model that isn’t wildly expensive
`balanced` (default)	0.40	0.35	0.25	The knee of the quality/cost curve
`cost_first`	0.15	0.70	0.15	The cheapest model that clears a quality floor
`latency_first`	0.15	0.25	0.60	The fastest model that clears a quality floor

Because each candidate’s present weights are renormalised, only the ratio between w_q, w_c and w_l affects the outcome, not the absolute sum. The table shows the shipped vectors; the gateway operator can adjust a profile’s vector without an API change.

Choosing a profile

Send weight_profile on a chat completions request:

curl
Header
Python

curl https://api.meshapi.ai/v1/chat/completions \
  -H "Authorization: Bearer rsk_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "weight_profile": "cost_first",
    "messages": [{"role": "user", "content": "Summarise this changelog in one line"}]
  }'

curl https://api.meshapi.ai/v1/chat/completions \
  -H "Authorization: Bearer rsk_YOUR_KEY" \
  -H "X-Mesh-Weight-Profile: latency_first" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "Summarise this changelog in one line"}]
  }'

The header exists for proxies and gateways that cannot rewrite a JSON body. When both are sent, the body field wins — it is the parameter you wrote, the header is something an intermediary may have injected.

import httpx

response = httpx.post(
    "https://api.meshapi.ai/v1/chat/completions",
    headers={"Authorization": "Bearer rsk_YOUR_KEY"},
    json={
        "model": "auto",
        "weight_profile": "cost_first",
        "messages": [{"role": "user", "content": "Summarise this changelog in one line"}],
    },
)

The Python SDK’s ChatCompletionParams has no weight_profile field and rejects unknown fields, so send the request directly as above, or set the profile on the key.

Precedence

The effective profile is resolved highest-first:

Request — the weight_profile body field, or the X-Mesh-Weight-Profile header
API key — the weight_profile entry in the key’s routing policy
Team / organization — the same entry on the key’s routing-policy template, else your org’s default template
Gateway default — balanced

So you can set a house default on the key and still override it for one request.

An unrecognised profile name on a request is not an error. A typo like "cheapest" in the body or header falls back to balanced and the request is served normally. A misspelt profile is silently the default rather than a 400. Check the spelling against the four names above if routing isn’t behaving the way you expect. On a key’s routing policy or a template, an unknown name is rejected with a 422 when you save it.

weight_profile applies to POST /v1/chat/completions only, and only while weighted is the algorithm serving the request. On /v1/responses and /v1/router/select it is ignored and the gateway default applies. Sending it is always safe — it is stripped before your request reaches the upstream provider.

Previewing a decision

POST /v1/router/select returns the model the Auto Router would pick, without running inference or billing you for one. Useful for pinning a model yourself, or for checking what a prompt classifies as.

curl https://api.meshapi.ai/v1/router/select \
  -H "Authorization: Bearer rsk_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Write a Python quicksort implementation"}]
  }'

{
  "model": "anthropic/claude-haiku-4.5",
  "auto_router": { "fallback_used": false, "fallback_reason": null, "router_version": "v10" },
  "reasoning_effort": "medium"
}

The routing-data version that made the pick is also returned in the X-Mesh-Router-Version header. The key needs the router:read scope. Optional request fields:

Field	Effect
`candidate_models`	Restricts the pick to ids you name, intersected with your key’s policy. An empty list or empty intersection is a `422`
`exclude_models`	Removes ids from consideration, which is how an “ask another model” flow avoids re-picking the one already shown. Excluding every remaining model is a `422`
`api_type`	`completions` (default) or `responses`: which surface’s pool to pick from
`router_version`	The routing-data version to pick on. See Auto Routing
`tags`	Your own labels for the request, as on inference requests

The classifier still runs, so a select call takes roughly as long as the routing stage of a real request. reasoning_effort is a hint: the benchmark and weighted algorithms set it from the classification, heuristic returns "none", and it is null otherwise.

Nothing here can fail a request

model: "auto" always resolves to a concrete model. Every layer is fail-soft by design:

An algorithm that declines, times out, or errors falls through to the next in the chain.
A chain that exhausts every algorithm falls through to the configured default model.
A missing scoring signal drops that term instead of dropping the candidate.
An unknown weight_profile on a request degrades to balanced.
A settings-store outage falls back to the built-in defaults rather than refusing to route.

Whenever the default model serves, or the router’s pick had to be replaced, the response says so: x_auto_routed_fallback and x_auto_routed_fallback_reason on the body, or X-Auto-Routed-Fallback and X-Auto-Routed-Fallback-Reason on a stream. A pick made by a later algorithm in the chain is not a fallback. See Fallback behavior for the reasons, and Debug → Auto Routing when a pick surprises you.

Introduction

Accounts & Access

Core Inference

Retrieval & Search

Multimodal

Operations

Mesh Code

Mesh CLI

Platform

Reference

Routing Algorithms

Two layers

The algorithms

What the pool is

Traffic weights

Scoring weights

The three signals

Missing signals degrade, they don’t error

Weight profiles

Choosing a profile

Precedence

Previewing a decision

Nothing here can fail a request

​Two layers

​The algorithms

​What the pool is

​Traffic weights

​Scoring weights

​The three signals

​Missing signals degrade, they don’t error

​Weight profiles

​Choosing a profile

​Precedence

​Previewing a decision

​Nothing here can fail a request

Two layers

The algorithms

What the pool is

Traffic weights

Scoring weights

The three signals

Missing signals degrade, they don’t error

Weight profiles

Choosing a profile

Precedence

Previewing a decision

Nothing here can fail a request