Streaming - Mesh API

Basic streaming

The streaming API differs by language: Python uses a separate stream() method, Node.js passes stream: true to create(), and Go returns channels.

Python
Node.js
Go

stream() is a separate method from create(). It returns a sync iterator.

from meshapi import MeshAPI, ChatCompletionParams, ChatMessage

client = MeshAPI(base_url="https://api.meshapi.ai", token="rsk_...")

for chunk in client.chat.completions.stream(
    ChatCompletionParams(
        model="openai/gpt-4o-mini",
        messages=[ChatMessage(role="user", content="Count exactly from 1 to 3.")],
    )
):
    if chunk.choices and chunk.choices[0].delta and chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="", flush=True)

Pass stream: true to the same create() method. It returns an AsyncIterable of chunks.

for await (const chunk of client.chat.completions.create({
  model: "openai/gpt-4o-mini",
  messages: [{ role: "user", content: "Count exactly from 1 to 3." }],
  stream: true,
})) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

Stream() returns two channels — one for chunks, one for the terminal error.

model := "openai/gpt-4o-mini"
maxTokens := 40
chunkCh, errCh := client.Chat.Completions.Stream(ctx, meshapi.ChatCompletionParams{
    Model:     &model,
    Messages:  []meshapi.ChatMessage{{Role: "user", Content: "Count from 1 to 5."}},
    MaxTokens: &maxTokens,
})
for chunk := range chunkCh {
    if len(chunk.Choices) > 0 && chunk.Choices[0].Delta != nil && chunk.Choices[0].Delta.Content != nil {
        fmt.Print(*chunk.Choices[0].Delta.Content)
    }
}
if err := <-errCh; err != nil {
    log.Fatal(err)
}

Async streaming (Python)

Use AsyncMeshAPI to stream in async contexts.

import asyncio
from meshapi import AsyncMeshAPI, ChatCompletionParams, ChatMessage

async def main():
    async with AsyncMeshAPI(base_url="https://api.meshapi.ai", token="rsk_...") as client:
        async for chunk in client.chat.completions.stream(
            ChatCompletionParams(
                model="openai/gpt-4o-mini",
                messages=[ChatMessage(role="user", content="Explain async/await.")],
            )
        ):
            if chunk.choices and chunk.choices[0].delta:
                print(chunk.choices[0].delta.content or "", end="", flush=True)

asyncio.run(main())

Cancelling a stream

Python
Node.js
Go

Break out of the iterator at any point — no cleanup needed.

for chunk in client.chat.completions.stream(params):
    print(chunk.choices[0].delta.content or "", end="")
    if should_stop():
        break

Pass an AbortSignal to cancel the request mid-stream.

const controller = new AbortController();
setTimeout(() => controller.abort(), 5_000); // cancel after 5s

try {
  for await (const chunk of client.chat.completions.create(
    { model: "openai/gpt-4o-mini", messages: [...], stream: true },
    { signal: controller.signal },
  )) {
    process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
  }
} catch (err) {
  if (err.name === "AbortError") {
    console.log("Stream cancelled.");
  }
}

Cancel the context to stop the stream.

ctx, cancel := context.WithCancel(context.Background())
defer cancel()

chunkCh, errCh := client.Chat.Completions.Stream(ctx, params)
received := 0
for range chunkCh {
    received++
    if received >= 3 {
        cancel() // stop the stream
        break
    }
}
for range chunkCh {} // drain
<-errCh              // may be context.Canceled or nil — both acceptable

Stream recovery (Python)

Streams do not automatically retry mid-stream. If the connection drops, a MeshAPIError with error_code="stream_interrupted" is raised. Catch it and restart:

from meshapi import MeshAPIError

try:
    for chunk in client.chat.completions.stream(params):
        process(chunk)
except MeshAPIError as e:
    if e.error_code == "stream_interrupted":
        # restart from scratch — no partial resume
        for chunk in client.chat.completions.stream(params):
            process(chunk)

Non-streaming create() requests retry automatically on 429, 502, 503, and 504. Streams never retry automatically — the client would have to re-generate content already received.

​Basic streaming

​Async streaming (Python)

​Cancelling a stream

​Stream recovery (Python)

Basic streaming

Async streaming (Python)

Cancelling a stream

Stream recovery (Python)