> ## Documentation Index
> Fetch the complete documentation index at: https://developers.meshapi.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Request Logs

> A record of every request your organization makes, readable from your own Grafana or pushed into your own Splunk, kept for 7 days.

Mesh keeps one record for each authenticated HTTP request your organization makes: the route, the
status we answered with, how long it took, the key that made it, and the caller's address. The
records follow OpenTelemetry's naming, so any tool that understands OpenTelemetry HTTP fields reads
them without mapping. They are kept for **7 days**, on a best-effort basis.

You can read them two ways:

| | Grafana (pull) | Splunk (push) |
| - | - | - |
| How | Add Mesh as a Loki data source in your own Grafana | Mesh sends each record to your Splunk HTTP Event Collector |
| Setup | Self-serve, with an admin key | On request: [contact us](#push-to-your-splunk) |
| History | The last 7 days, queried on demand | From the moment it is switched on, live |
| Request and response bodies | Separate endpoint, separate permission | Never sent |

Every read is scoped to your organization by the key you connect with. Nothing in a query can name
another organization, and another organization's records never come back.

***

## What a record holds

One record per request, with the line `http_request`. Grafana shows the fields with underscores
(`http_route`); Splunk keeps OpenTelemetry's dots (`http.route`).

| Field (Grafana) | Field (Splunk) | Meaning |
| - | - | - |
| `http_request_method` | `http.request.method` | `GET`, `POST`, ... (`_OTHER` for an unusual method, with the raw one in `http_request_method_original`, in Splunk `http.request.method_original`) |
| `http_route` | `http.route` | The route template, for example `/v1/chat/completions` or `/v1/keys/{key_id}` |
| `url_path` | `url.path` | The path as called |
| `http_response_status_code` | `http.response.status_code` | The status we answered with |
| `latency_ms` | `latency_ms` | Time until we sent the response headers. On a streamed response that is time to first byte, not how long the stream ran |
| `total_duration_ms` | Not sent | Time until the response body finished. On a request that is not streamed it equals `latency_ms` |
| `request_id` | `request_id` | Our `req_...` id: the same as the `X-Request-Id` response header and the id on your [usage events](https://developers.meshapi.ai/docs/reference/usage-api.md) |
| `key_id`, `org_id`, `owner` | same | The key that made the request, your organization, and the user or key owner behind it |
| `client_address`, `user_agent_original` | `client.address`, `user_agent.original` | The caller's IP address and user agent |
| `trace_id`, `span_id` | same | Mesh's trace identifiers for the request |

Request and response bodies are not in the record; see [Request and response bodies](#request-and-response-bodies).

### Requests that have no record

* A request refused before its credential was read: no key, an invalid or revoked key, an unknown
  path, a body over the size limit or not valid JSON, or an unsupported `X-Mesh-Version`.
* A request made with another organization's key.
* WebSocket sessions (`/v1/realtime`, `/v1/live` and the streaming audio sockets).
* A request that failed with an unexpected server error before we answered it. An alert on `5xx`
  built only on these records will miss those.
* A streamed request whose server process stopped before the stream finished.
* Records dropped while our log pipeline is overloaded or down.
* Your own calls to the logs API.

***

## Read your logs in Grafana

Mesh serves a Loki-compatible read API at `https://api.meshapi.ai/v1/logs`. Grafana's Loki data
source, Explore view and Logs Drilldown app work against it unchanged.

**1. Create an admin key**

In **Dashboard → Settings → Admin keys**, create a key with:

* the permission `logs:read`, and nothing else unless the key needs it;
* scope `org`. `logs:read` covers the whole organization, so a narrower scope is refused;
* an expiry. Admin keys live at most 90 days.

Anyone holding `keys:write` and `logs:read` can create one (owners and admins by default). A key
can only carry permissions its creator holds, and it keeps
them until it expires or is revoked, even after its creator leaves the organization. See
[Admin Keys](https://developers.meshapi.ai/docs/getting-started/admin-keys.md).

Inference keys (`rsk_...`) are refused here with a `401`.

**2. Add the data source**

In Grafana, open **Connections → Data sources → Add data source → Loki** and set:

| Setting | Value |
| - | - |
| URL | `https://api.meshapi.ai/v1/logs` |
| Authentication | No authentication |
| HTTP headers | Header `Authorization`, value `Bearer mak_...` |
| Alerting: Manage alerts via Alerting UI | Off |

Basic auth is not supported: put the key in the `Authorization` header.

**3. Save & test**

Grafana answers **Data source successfully connected.** Open **Explore**, pick the data source,
and run `{service_name="meshapi-gateway"}`.

To provision the same data source from a file:

```yaml theme={null}
apiVersion: 1
datasources:
  - name: Mesh logs
    type: loki
    url: https://api.meshapi.ai/v1/logs
    jsonData:
      httpHeaderName1: Authorization
      manageAlerts: false
    secureJsonData:
      httpHeaderValue1: Bearer $MESH_ADMIN_KEY
```

### Queries

The stream labels are `service_name` (`meshapi-gateway`) and `deployment_environment_name`. Every
field in the table above can be filtered with `| field="value"` or compared with `| field >= n`.

```
# every request
{service_name="meshapi-gateway"}

# only errors
{service_name="meshapi-gateway"} | http_response_status_code >= 400

# one request, by the X-Request-Id you got back
{service_name="meshapi-gateway"} | request_id="req_01..."

# slow requests on one route
{service_name="meshapi-gateway"} | http_route="/v1/chat/completions" | latency_ms > 5000

# requests per route, as a time series
sum by (http_route) (count_over_time({service_name="meshapi-gateway"} [5m]))
```

> [!NOTE]
>
> A bare `req_...` typed into Grafana's search box matches the line text only, which is `http_request`.
> Filter on the field with `| request_id="req_..."` instead.

### Query the API directly

The same API answers plain HTTP, with Loki's request parameters and JSON responses:

```bash theme={null}
curl -G "https://api.meshapi.ai/v1/logs/loki/api/v1/query_range" \
  -H "Authorization: Bearer $MESH_ADMIN_KEY" \
  --data-urlencode 'query={service_name="meshapi-gateway"} | http_response_status_code >= 400' \
  --data-urlencode 'since=1h' \
  --data-urlencode 'limit=100'
```

The paths served are the ones Grafana and Logs Drilldown read: `query`, `query_range`, `labels`,
`label/{name}/values`, `series`, `detected_fields`, `detected_field/{name}/values`,
`detected_labels`, `index/stats`, `index/volume`, `index/volume_range`, `patterns`,
`format_query` and `status/buildinfo`, each under `/v1/logs/loki/api/v1/`. Anything else is a `404`.
The [Admin API reference](https://developers.meshapi.ai/docs/reference/admin-api.md#logs) lists each one with its permission.

***

## Request and response bodies

Bodies are never in the request record and never sent to Splunk. Where request body logging is on
for a key (or as your organization's default, see
[Organizations](https://developers.meshapi.ai/docs/getting-started/organizations.md)), an admin key with `logs:read_bodies` reads
the stored bodies:

```bash theme={null}
curl -G "https://api.meshapi.ai/v1/logs/bodies" \
  -H "Authorization: Bearer $MESH_ADMIN_KEY" \
  --data-urlencode "request_id=req_01..."
```

`logs:read_bodies` is separate from `logs:read` and never implied by it. Only owners and admins
hold it, and no custom role can carry it.

| Parameter | Meaning |
| - | - |
| `request_id` | Only this request (`req_` and 26 characters) |
| `key_id` | Only requests made with this API key |
| `since`, `until` | The window, at most 7 days apart. Default: the last hour. A time with no zone is UTC |
| `limit` | 1 to 50 records, default 20, newest first |
| `cursor` | The `next_cursor` of the previous page |

The answer is `{"data": [...], "next_cursor": ...}`. Each record has `timestamp`, `request_id`,
`key_id`, `request` and `response`; a direction your body-logging setting does not keep is `null`.
When `next_cursor` is set, ask again with it as `cursor` and the same `since`, `until` and filters
for the next, older page. Pass `until` (and `since`) on every page with the values of the first,
or the window moves with the clock while you page. A page can hold fewer than `limit` records, or
none, while `next_cursor` is still set.

Each of `request` and `response` is `{"content": ..., "truncated": false, "bytes": n}`, or
`{"text": "...", "truncated": true, "bytes": n, "stored_bytes": m}` when the body was cut (a text
body that was not cut has `text` and `"truncated": false`). A body too large to store at all is
`{"truncated": true, "bytes": n, "stored_bytes": 0}`. `bytes` is the size of the stored body
(after redaction and scrubbing) and `stored_bytes` how much of it is returned.

* **Every read is in your audit log first** (`log_bodies.read`, with the filters and window),
  before anything is fetched. If the audit log cannot be written, the answer is `503` and nothing
  is read.
* **Zero data retention:** while it is on for your organization, the answer is `403` and nothing is
  read, including bodies stored before it was turned on.
* **Your setting is read again on every read.** Turn body logging off for a key, or narrow it to
  one direction, and its bodies stop being served within a minute, including ones already stored.
* Bodies are the stored copy. Request bodies are PII-redacted where a redaction policy is in
  force; responses are not. Credentials are scrubbed, except in a streamed response, which is kept
  as sent. Each body is cut at 32 KiB. A large image or audio payload may be replaced by a reference
  to where Mesh stored it, which is not a download link.

***

## Push to your Splunk

Mesh can send each request record to your own Splunk through its HTTP Event Collector (HEC), a few
seconds after the request. Bodies and every other line stay out. Setup is done by Mesh for your
organization: email **[contact@meshapi.ai](mailto:contact@meshapi.ai)** with the subject **Splunk logs**.

**1. Create an index and a token**

Create an index for Mesh, for example `mesh`. Then create a HEC token (**Settings → Data inputs →
HTTP Event Collector → New Token**) whose allowed index is that index and nothing else, with
indexer acknowledgement off.

**2. Check your HEC's certificate**

Your HEC must serve `https` with a certificate that names its hostname (as a subject alternative
name), issued by a public CA or by your own CA. For your own CA, send us that CA's certificate.
Splunk Cloud's certificate already qualifies. Splunk Enterprise's out-of-the-box certificate names
no host, so we cannot verify it: replace it before we connect.

**3. Send us the details**

* the HEC URL, for example `https://splunk.example.com:8088/services/collector` (Splunk Cloud:
  `https://http-inputs-<stack>.splunkcloud.com:443/services/collector`);
* the index name, and the CA certificate if the previous step needs it;
* the token, through the secure channel we give you for it. Never in an email, ticket or chat.

If your HEC only accepts allowlisted source addresses, ask us for ours.

### What arrives

| Splunk | Value |
| - | - |
| `index` | Yours |
| `source` | `meshapi` |
| `sourcetype` | `meshapi:http_request` |
| `_time` | When the record was written: as the response headers went out, or when a streamed body finished |
| event text | `http_request` |

The fields are the ones in [What a record holds](#what-a-record-holds), with dots, plus
`otel.log.severity.text` (`WARNING` for a status of 400 or above, else `INFO`) and a few fields
describing the Mesh process that wrote the record (`service.*`, `deployment.environment.name`,
`telemetry.sdk.*`).

They are index-time fields: match a value with `field::value` (a plain `field=value` search finds
nothing unless your `fields.conf` marks the field `INDEXED = true`), and compare numbers with
`where`:

```
index=mesh sourcetype="meshapi:http_request"
index=mesh request_id::req_01...
index=mesh sourcetype="meshapi:http_request" | where 'http.response.status_code' >= 400
index=mesh sourcetype="meshapi:http_request" | stats count by http.route
```

### Delivery

* **Live only.** Records are pushed as they are written. While your HEC is unreachable or busy,
  each batch is retried for 5 minutes and then dropped. A batch your HEC refuses (a wrong or
  deleted token, a wrong index) is dropped at once. Dropped records are not resent; read them
  through [Grafana](#read-your-logs-in-grafana) while they are inside the 7 days.
* Records reach your HEC within a few seconds, plus Splunk's own indexing time.
* **Rotating the token:** create a new token, send it to us the same way, and delete the old one
  once we confirm the switch. Keep both until then. The switch can drop a few seconds of records.
* **Stopping:** ask us to remove the push, then delete the token. Deleting the token alone does not
  stop us sending; every batch is refused until the push is removed.

***

## Limits

* **7 days.** Older records are gone.
* **600 requests a minute per organization**, across all your keys and dashboards. Past it the
  answer is `429` with `Retry-After`. A dashboard makes one call per panel per refresh, so a
  10-second refresh on a large dashboard can reach it.
* **32 MiB per answer.** A larger answer is refused with `422 log_result_too_large`, never cut short.
  Narrow the time range or the query; a 5,000-line log query is far under it.
* The log store's own query limits apply on top: time range, entries per query, series.
* **Not served:** live tail (Grafana's **Live** button), log push, deletes, and alert or recording
  rules stored in Loki. Grafana-managed alerts that run ordinary queries work.

## Errors

| Status | Meaning |
| - | - |
| `400` | The query was refused, for example a LogQL syntax error: the body is the log store's own message. For bodies: `since` is not before `until`, or they are more than 7 days apart |
| `401` | No key, not an admin key, or a revoked or expired one |
| `403` | The key lacks `logs:read` (or `logs:read_bodies` for bodies), or its scope is narrower than `org`; for bodies, zero data retention is on |
| `404` | A path this API does not serve |
| `422` | `log_result_too_large`: the answer is over 32 MiB. For bodies: a `request_id`, `key_id`, `since`, `until`, `limit` or `cursor` that is not valid |
| `429` | Over the per-organization limit: wait `Retry-After` seconds. A `429` without `Retry-After` means the log store's query queue is full: back off and retry |
| `503` | The log store is unavailable (`log_store_unavailable`), or for bodies the audit log or the retention check is (`audit_store_unavailable`, `zdr_state_unavailable`). Retry later |

See [Errors](https://developers.meshapi.ai/docs/reference/errors.md) for the envelope.

## Rotating the key

Admin keys expire. Rotate yours in the dashboard at least a day before it does: the old key keeps
working through a grace window (24 hours by default, up to 7 days) that never runs past its own
expiry. Paste the new key into the data source's
`Authorization` header within that window. After expiry Grafana shows the `401` as a data source
error.

## Personal data

These records hold your callers' IP addresses and user agents, and the `owner` id behind each
request. This log copy is deleted when it ages out after 7 days; an end-user erasure request does
not reach it sooner. Once records are in your Splunk, their retention and erasure are yours.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.