Skip to main content
The Mesh API is more than just a proxy; it’s a high-performance orchestration layer that manages dozens of AI providers, monitors costs, and ensures maximum availability.

​
High-Level Workflow

When you call /v1/chat/completions, the Mesh Router performs the following tasks:
  1. Authentication: Validates your rsk_ key, checks for active balance, and verifies spend caps.
  2. Provider Selection: (If multiple providers offer the same model) Identifies the provider with the lowest current latency and cost.
  3. Secret Retrieval: Fetches your encrypted provider keys from GCP Secret Manager (with memory-caching for sub-millisecond lookups).
  4. Request Transformation: Converts your request into the provider-specific format (e.g., Anthropic Messages API or Google Vertex AI).
  5. Streaming/Buffering: Pipes the response back to your client in real-time (supporting Server-Sent Events).
  6. Usage Logging: Records the total tokens consumed, calculates the final cost in USD, and updates your balance atomically.

​
Resilience Features

  • Automatic Failover: If an upstream provider returns a transient error, Mesh retries and then re-sends the request for the same model through another provider that serves it. This is on by default for every key. See Retry & Fallback.
  • Circuit Breaking: Mesh monitors error rates for each model-provider pair. If a provider starts failing, it’s temporarily removed from the routing pool.
  • Regional Optimization: To minimize latency, Mesh routes requests to the nearest cloud region for global providers (Bedrock, Vertex AI).

​
Security & Privacy

  • Key Scoping: API keys are isolated per owner. Your keys can never be used by another account.
  • Request logs: By default, Mesh stores the request and response bodies of each API call in its request logs, next to the metadata (tokens, model, latency, cost). With PII redaction on, the stored request body is the redacted copy. The bodies Mesh sends to upstream providers are kept for 14 days.
  • Zero data retention: An org owner or admin can turn it on under Settings → Data retention. Mesh then stores no request or response bodies for any key in the org, including the copies sent upstream, and keeps only the metadata. See Organizations.
  • Encrypted Secrets: All upstream keys are stored in a hardware-isolated GCP Secret Manager with restricted IAM access.