Skip to main content
Mesh keeps one record for each authenticated HTTP request your organization makes: the route, the status we answered with, how long it took, the key that made it, and the caller’s address. The records follow OpenTelemetry’s naming, so any tool that understands OpenTelemetry HTTP fields reads them without mapping. They are kept for 7 days, on a best-effort basis. You can read them two ways: Every read is scoped to your organization by the key you connect with. Nothing in a query can name another organization, and another organization’s records never come back.

What a record holds

One record per request, with the line http_request. Grafana shows the fields with underscores (http_route); Splunk keeps OpenTelemetry’s dots (http.route). Request and response bodies are not in the record; see Request and response bodies.

Requests that have no record

  • A request refused before its credential was read: no key, an invalid or revoked key, an unknown path, a body over the size limit or not valid JSON, or an unsupported X-Mesh-Version.
  • A request made with another organization’s key.
  • WebSocket sessions (/v1/realtime, /v1/live and the streaming audio sockets).
  • A request that failed with an unexpected server error before we answered it. An alert on 5xx built only on these records will miss those.
  • A streamed request whose server process stopped before the stream finished.
  • Records dropped while our log pipeline is overloaded or down.
  • Your own calls to the logs API.

Read your logs in Grafana

Mesh serves a Loki-compatible read API at https://api.meshapi.ai/v1/logs. Grafana’s Loki data source, Explore view and Logs Drilldown app work against it unchanged.
1

Create an admin key

In Dashboard → Settings → Admin keys, create a key with:
  • the permission logs:read, and nothing else unless the key needs it;
  • scope org. logs:read covers the whole organization, so a narrower scope is refused;
  • an expiry. Admin keys live at most 90 days.
Anyone holding keys:write and logs:read can create one (owners and admins by default). A key can only carry permissions its creator holds, and it keeps them until it expires or is revoked, even after its creator leaves the organization. See Admin Keys.Inference keys (rsk_...) are refused here with a 401.
2

Add the data source

In Grafana, open Connections → Data sources → Add data source → Loki and set:Basic auth is not supported: put the key in the Authorization header.
3

Save & test

Grafana answers Data source successfully connected. Open Explore, pick the data source, and run {service_name="meshapi-gateway"}.
To provision the same data source from a file:

Queries

The stream labels are service_name (meshapi-gateway) and deployment_environment_name. Every field in the table above can be filtered with | field="value" or compared with | field >= n.
A bare req_... typed into Grafana’s search box matches the line text only, which is http_request. Filter on the field with | request_id="req_..." instead.

Query the API directly

The same API answers plain HTTP, with Loki’s request parameters and JSON responses:
The paths served are the ones Grafana and Logs Drilldown read: query, query_range, labels, label/{name}/values, series, detected_fields, detected_field/{name}/values, detected_labels, index/stats, index/volume, index/volume_range, patterns, format_query and status/buildinfo, each under /v1/logs/loki/api/v1/. Anything else is a 404. The Admin API reference lists each one with its permission.

Request and response bodies

Bodies are never in the request record and never sent to Splunk. Where request body logging is on for a key (or as your organization’s default, see Organizations), an admin key with logs:read_bodies reads the stored bodies:
logs:read_bodies is separate from logs:read and never implied by it. Only owners and admins hold it, and no custom role can carry it. The answer is {"data": [...], "next_cursor": ...}. Each record has timestamp, request_id, key_id, request and response; a direction your body-logging setting does not keep is null. When next_cursor is set, ask again with it as cursor and the same since, until and filters for the next, older page. Pass until (and since) on every page with the values of the first, or the window moves with the clock while you page. A page can hold fewer than limit records, or none, while next_cursor is still set. Each of request and response is {"content": ..., "truncated": false, "bytes": n}, or {"text": "...", "truncated": true, "bytes": n, "stored_bytes": m} when the body was cut (a text body that was not cut has text and "truncated": false). A body too large to store at all is {"truncated": true, "bytes": n, "stored_bytes": 0}. bytes is the size of the stored body (after redaction and scrubbing) and stored_bytes how much of it is returned.
  • Every read is in your audit log first (log_bodies.read, with the filters and window), before anything is fetched. If the audit log cannot be written, the answer is 503 and nothing is read.
  • Zero data retention: while it is on for your organization, the answer is 403 and nothing is read, including bodies stored before it was turned on.
  • Your setting is read again on every read. Turn body logging off for a key, or narrow it to one direction, and its bodies stop being served within a minute, including ones already stored.
  • Bodies are the stored copy. Request bodies are PII-redacted where a redaction policy is in force; responses are not. Credentials are scrubbed, except in a streamed response, which is kept as sent. Each body is cut at 32 KiB. A large image or audio payload may be replaced by a reference to where Mesh stored it, which is not a download link.

Push to your Splunk

Mesh can send each request record to your own Splunk through its HTTP Event Collector (HEC), a few seconds after the request. Bodies and every other line stay out. Setup is done by Mesh for your organization: email contact@meshapi.ai with the subject Splunk logs.
1

Create an index and a token

Create an index for Mesh, for example mesh. Then create a HEC token (Settings → Data inputs → HTTP Event Collector → New Token) whose allowed index is that index and nothing else, with indexer acknowledgement off.
2

Check your HEC's certificate

Your HEC must serve https with a certificate that names its hostname (as a subject alternative name), issued by a public CA or by your own CA. For your own CA, send us that CA’s certificate. Splunk Cloud’s certificate already qualifies. Splunk Enterprise’s out-of-the-box certificate names no host, so we cannot verify it: replace it before we connect.
3

Send us the details

  • the HEC URL, for example https://splunk.example.com:8088/services/collector (Splunk Cloud: https://http-inputs-<stack>.splunkcloud.com:443/services/collector);
  • the index name, and the CA certificate if the previous step needs it;
  • the token, through the secure channel we give you for it. Never in an email, ticket or chat.
If your HEC only accepts allowlisted source addresses, ask us for ours.

What arrives

The fields are the ones in What a record holds, with dots, plus otel.log.severity.text (WARNING for a status of 400 or above, else INFO) and a few fields describing the Mesh process that wrote the record (service.*, deployment.environment.name, telemetry.sdk.*). They are index-time fields: match a value with field::value (a plain field=value search finds nothing unless your fields.conf marks the field INDEXED = true), and compare numbers with where:

Delivery

  • Live only. Records are pushed as they are written. While your HEC is unreachable or busy, each batch is retried for 5 minutes and then dropped. A batch your HEC refuses (a wrong or deleted token, a wrong index) is dropped at once. Dropped records are not resent; read them through Grafana while they are inside the 7 days.
  • Records reach your HEC within a few seconds, plus Splunk’s own indexing time.
  • Rotating the token: create a new token, send it to us the same way, and delete the old one once we confirm the switch. Keep both until then. The switch can drop a few seconds of records.
  • Stopping: ask us to remove the push, then delete the token. Deleting the token alone does not stop us sending; every batch is refused until the push is removed.

Limits

  • 7 days. Older records are gone.
  • 600 requests a minute per organization, across all your keys and dashboards. Past it the answer is 429 with Retry-After. A dashboard makes one call per panel per refresh, so a 10-second refresh on a large dashboard can reach it.
  • 32 MiB per answer. A larger answer is refused with 422 log_result_too_large, never cut short. Narrow the time range or the query; a 5,000-line log query is far under it.
  • The log store’s own query limits apply on top: time range, entries per query, series.
  • Not served: live tail (Grafana’s Live button), log push, deletes, and alert or recording rules stored in Loki. Grafana-managed alerts that run ordinary queries work.

Errors

See Errors for the envelope.

Rotating the key

Admin keys expire. Rotate yours in the dashboard at least a day before it does: the old key keeps working through a grace window (24 hours by default, up to 7 days) that never runs past its own expiry. Paste the new key into the data source’s Authorization header within that window. After expiry Grafana shows the 401 as a data source error.

Personal data

These records hold your callers’ IP addresses and user agents, and the owner id behind each request. This log copy is deleted when it ages out after 7 days; an end-user erasure request does not reach it sooner. Once records are in your Splunk, their retention and erasure are yours.