Realtime Audio

View as Markdown

Realtime Audio

client.realtime opens a bidirectional WebSocket session to wss://api.meshapi.ai/v1/realtime. The wire format is identical to OpenAI’s Realtime API.

Node 22+ has WebSocket built in. On Node 18–21, install the ws package: npm install ws.

Connect

1import { MeshAPI, RealtimeError } from "meshapi-node-sdk";
2
3const client = new MeshAPI({ baseUrl: "https://api.meshapi.ai", token: "rsk_..." });
4
5const session = await client.realtime.connect({ model: "openai/gpt-realtime-mini" });

Configure the session

1await session.send({
2 type: "session.update",
3 session: {
4 type: "realtime",
5 modalities: ["audio", "text"],
6 voice: "alloy",
7 instructions: "You are a helpful assistant.",
8 },
9});

Send audio

1// pcmBytes is a Uint8Array of raw 16-bit PCM at 24 kHz mono
2await session.sendAudio(pcmBytes);

Async iterator

The async iterator yields RealtimeMessage objects and throws RealtimeError on server error envelopes:

1try {
2 for await (const msg of session) {
3 if (msg.audio) {
4 playAudio(msg.audio); // Uint8Array
5 } else if (msg.event) {
6 const type = msg.event.type as string;
7 if (type === "response.text.delta") {
8 process.stdout.write(msg.event.delta as string);
9 } else if (type === "response.done") {
10 break;
11 }
12 }
13 }
14} finally {
15 await session.close();
16}

Callback style

Use on() when you prefer event handlers over iteration:

1session
2 .on("message", (msg) => {
3 if (msg.audio) playAudio(msg.audio);
4 else if (msg.event?.type === "response.text.delta") {
5 process.stdout.write(msg.event.delta as string);
6 }
7 })
8 .on("error", (err) => console.error("error:", err))
9 .on("close", (code, reason) => console.log("closed", code, reason));

Close

close() returns a Promise that resolves when the TCP close handshake completes:

1await session.close();

Error handling

1try {
2 for await (const msg of session) { ... }
3} catch (err) {
4 if (err instanceof RealtimeError) {
5 console.error("code:", err.code); // "invalid_api_key", "insufficient_quota", …
6 console.error("message:", err.message);
7 }
8}

Full voice agent example

1const session = await client.realtime.connect({ model: "openai/gpt-realtime-mini" });
2
3await session.send({
4 type: "session.update",
5 session: { type: "realtime", modalities: ["audio"], voice: "alloy" },
6});
7
8// Stream microphone audio
9(async () => {
10 for await (const chunk of micStream) {
11 await session.sendAudio(chunk);
12 }
13 await session.send({ type: "input_audio_buffer.commit" });
14 await session.send({ type: "response.create" });
15})();
16
17// Play response audio
18for await (const msg of session) {
19 if (msg.audio) speaker.write(msg.audio);
20}
21await session.close();

Supported models

Model ID
openai/gpt-realtime-mini
openai/gpt-realtime
openai/gpt-realtime-2