Every read is scoped to your organization by the key you connect with. Nothing in a query can name
another organization, and another organization’s records never come back.
What a record holds
One record per request, with the linehttp_request. Grafana shows the fields with underscores
(http_route); Splunk keeps OpenTelemetry’s dots (http.route).
Request and response bodies are not in the record; see Request and response bodies.
Requests that have no record
- A request refused before its credential was read: no key, an invalid or revoked key, an unknown
path, a body over the size limit or not valid JSON, or an unsupported
X-Mesh-Version. - A request made with another organization’s key.
- WebSocket sessions (
/v1/realtime,/v1/liveand the streaming audio sockets). - A request that failed with an unexpected server error before we answered it. An alert on
5xxbuilt only on these records will miss those. - A streamed request whose server process stopped before the stream finished.
- Records dropped while our log pipeline is overloaded or down.
- Your own calls to the logs API.
Read your logs in Grafana
Mesh serves a Loki-compatible read API athttps://api.meshapi.ai/v1/logs. Grafana’s Loki data
source, Explore view and Logs Drilldown app work against it unchanged.
1
Create an admin key
In Dashboard → Settings → Admin keys, create a key with:
- the permission
logs:read, and nothing else unless the key needs it; - scope
org.logs:readcovers the whole organization, so a narrower scope is refused; - an expiry. Admin keys live at most 90 days.
keys:write and logs:read can create one (owners and admins by default). A key
can only carry permissions its creator holds, and it keeps
them until it expires or is revoked, even after its creator leaves the organization. See
Admin Keys.Inference keys (rsk_...) are refused here with a 401.2
Add the data source
In Grafana, open Connections → Data sources → Add data source → Loki and set:
Basic auth is not supported: put the key in the
Authorization header.3
Save & test
Grafana answers Data source successfully connected. Open Explore, pick the data source,
and run
{service_name="meshapi-gateway"}.Queries
The stream labels areservice_name (meshapi-gateway) and deployment_environment_name. Every
field in the table above can be filtered with | field="value" or compared with | field >= n.
A bare
req_... typed into Grafana’s search box matches the line text only, which is http_request.
Filter on the field with | request_id="req_..." instead.Query the API directly
The same API answers plain HTTP, with Loki’s request parameters and JSON responses:query, query_range, labels,
label/{name}/values, series, detected_fields, detected_field/{name}/values,
detected_labels, index/stats, index/volume, index/volume_range, patterns,
format_query and status/buildinfo, each under /v1/logs/loki/api/v1/. Anything else is a 404.
The Admin API reference lists each one with its permission.
Request and response bodies
Bodies are never in the request record and never sent to Splunk. Where request body logging is on for a key (or as your organization’s default, see Organizations), an admin key withlogs:read_bodies reads
the stored bodies:
logs:read_bodies is separate from logs:read and never implied by it. Only owners and admins
hold it, and no custom role can carry it.
The answer is
{"data": [...], "next_cursor": ...}. Each record has timestamp, request_id,
key_id, request and response; a direction your body-logging setting does not keep is null.
When next_cursor is set, ask again with it as cursor and the same since, until and filters
for the next, older page. Pass until (and since) on every page with the values of the first,
or the window moves with the clock while you page. A page can hold fewer than limit records, or
none, while next_cursor is still set.
Each of request and response is {"content": ..., "truncated": false, "bytes": n}, or
{"text": "...", "truncated": true, "bytes": n, "stored_bytes": m} when the body was cut (a text
body that was not cut has text and "truncated": false). A body too large to store at all is
{"truncated": true, "bytes": n, "stored_bytes": 0}. bytes is the size of the stored body
(after redaction and scrubbing) and stored_bytes how much of it is returned.
- Every read is in your audit log first (
log_bodies.read, with the filters and window), before anything is fetched. If the audit log cannot be written, the answer is503and nothing is read. - Zero data retention: while it is on for your organization, the answer is
403and nothing is read, including bodies stored before it was turned on. - Your setting is read again on every read. Turn body logging off for a key, or narrow it to one direction, and its bodies stop being served within a minute, including ones already stored.
- Bodies are the stored copy. Request bodies are PII-redacted where a redaction policy is in force; responses are not. Credentials are scrubbed, except in a streamed response, which is kept as sent. Each body is cut at 32 KiB. A large image or audio payload may be replaced by a reference to where Mesh stored it, which is not a download link.
Push to your Splunk
Mesh can send each request record to your own Splunk through its HTTP Event Collector (HEC), a few seconds after the request. Bodies and every other line stay out. Setup is done by Mesh for your organization: email contact@meshapi.ai with the subject Splunk logs.1
Create an index and a token
Create an index for Mesh, for example
mesh. Then create a HEC token (Settings → Data inputs →
HTTP Event Collector → New Token) whose allowed index is that index and nothing else, with
indexer acknowledgement off.2
Check your HEC's certificate
Your HEC must serve
https with a certificate that names its hostname (as a subject alternative
name), issued by a public CA or by your own CA. For your own CA, send us that CA’s certificate.
Splunk Cloud’s certificate already qualifies. Splunk Enterprise’s out-of-the-box certificate names
no host, so we cannot verify it: replace it before we connect.3
Send us the details
- the HEC URL, for example
https://splunk.example.com:8088/services/collector(Splunk Cloud:https://http-inputs-<stack>.splunkcloud.com:443/services/collector); - the index name, and the CA certificate if the previous step needs it;
- the token, through the secure channel we give you for it. Never in an email, ticket or chat.
What arrives
The fields are the ones in What a record holds, with dots, plus
otel.log.severity.text (WARNING for a status of 400 or above, else INFO) and a few fields
describing the Mesh process that wrote the record (service.*, deployment.environment.name,
telemetry.sdk.*).
They are index-time fields: match a value with field::value (a plain field=value search finds
nothing unless your fields.conf marks the field INDEXED = true), and compare numbers with
where:
Delivery
- Live only. Records are pushed as they are written. While your HEC is unreachable or busy, each batch is retried for 5 minutes and then dropped. A batch your HEC refuses (a wrong or deleted token, a wrong index) is dropped at once. Dropped records are not resent; read them through Grafana while they are inside the 7 days.
- Records reach your HEC within a few seconds, plus Splunk’s own indexing time.
- Rotating the token: create a new token, send it to us the same way, and delete the old one once we confirm the switch. Keep both until then. The switch can drop a few seconds of records.
- Stopping: ask us to remove the push, then delete the token. Deleting the token alone does not stop us sending; every batch is refused until the push is removed.
Limits
- 7 days. Older records are gone.
- 600 requests a minute per organization, across all your keys and dashboards. Past it the
answer is
429withRetry-After. A dashboard makes one call per panel per refresh, so a 10-second refresh on a large dashboard can reach it. - 32 MiB per answer. A larger answer is refused with
422 log_result_too_large, never cut short. Narrow the time range or the query; a 5,000-line log query is far under it. - The log store’s own query limits apply on top: time range, entries per query, series.
- Not served: live tail (Grafana’s Live button), log push, deletes, and alert or recording rules stored in Loki. Grafana-managed alerts that run ordinary queries work.
Errors
See Errors for the envelope.
Rotating the key
Admin keys expire. Rotate yours in the dashboard at least a day before it does: the old key keeps working through a grace window (24 hours by default, up to 7 days) that never runs past its own expiry. Paste the new key into the data source’sAuthorization header within that window. After expiry Grafana shows the 401 as a data source
error.
Personal data
These records hold your callers’ IP addresses and user agents, and theowner id behind each
request. This log copy is deleted when it ages out after 7 days; an end-user erasure request does
not reach it sooner. Once records are in your Splunk, their retention and erasure are yours.