Realtime Audio

View as Markdown

Realtime Audio

client.Realtime opens a bidirectional WebSocket session to wss://api.meshapi.ai/v1/realtime. The wire format is identical to OpenAI’s Realtime API — every event you send and receive is shaped exactly as upstream documents it.

Connect and close

1session, err := client.Realtime.Connect(ctx, meshapi.RealtimeConnectParams{
2 Model: "openai/gpt-realtime-2",
3})
4if err != nil {
5 log.Fatal(err)
6}
7defer session.Close()

Configure the session

1err = session.Send(ctx, map[string]any{
2 "type": "session.update",
3 "session": map[string]any{
4 "type": "realtime",
5 "output_modalities": []string{"audio"}, // or []string{"text"}
6 "instructions": "You are a helpful assistant.",
7 "audio": map[string]any{
8 "input": map[string]any{"format": map[string]any{"type": "audio/pcm", "rate": 24000}},
9 "output": map[string]any{"format": map[string]any{"type": "audio/pcm", "rate": 24000}, "voice": "alloy"},
10 },
11 },
12})

Send audio

1// pcmBytes is raw 16-bit PCM at 24 kHz mono
2err = session.SendAudio(ctx, pcmBytes)

RealtimeMessage

Every frame from the server is a meshapi.RealtimeMessage. Exactly one of the fields below is non-zero per message:

FieldTypeWhen set
Audio[]byteDecoded raw PCM audio, set on response.output_audio.delta events
TextstringRaw JSON string of the server event
Eventmap[string]anyParsed JSON map of the event (check Event["type"])

Output audio arrives in-band as base64 inside response.output_audio.delta events; the SDK decodes it into Audio, so those frames populate both Audio and Event. Check len(msg.Audio) > 0 before switching on Event["type"].

Receive frames

Receive blocks until the next frame arrives. Context cancellation unblocks it immediately:

1msg, err := session.Receive(ctx)
2if err == io.EOF {
3 fmt.Println("session closed")
4} else if err != nil {
5 log.Fatal(err)
6}
7
8if msg.Audio != nil {
9 // binary audio frame — play or buffer
10 playAudio(msg.Audio)
11} else if msg.Event != nil {
12 fmt.Println("event type:", msg.Event["type"])
13}

Events channel (concurrent pump)

For concurrent send/receive, use Events to pump frames into a channel:

1msgCh, errCh := session.Events(ctx)
2
3for msg := range msgCh {
4 switch {
5 case msg.Audio != nil:
6 playAudio(msg.Audio)
7 case msg.Event != nil:
8 switch msg.Event["type"] {
9 case "response.output_text.delta":
10 fmt.Print(msg.Event["delta"])
11 case "response.done":
12 fmt.Println("\n[done]")
13 case "error":
14 log.Printf("server error: %v", msg.Event)
15 }
16 }
17}
18
19if err := <-errCh; err != nil {
20 log.Fatal(err)
21}

Full voice agent example

1session, _ := client.Realtime.Connect(ctx, meshapi.RealtimeConnectParams{
2 Model: "openai/gpt-realtime-2",
3})
4defer session.Close()
5
6// Configure
7session.Send(ctx, map[string]any{
8 "type": "session.update",
9 "session": map[string]any{
10 "type": "realtime",
11 "output_modalities": []string{"audio"},
12 "audio": map[string]any{
13 "input": map[string]any{"format": map[string]any{"type": "audio/pcm", "rate": 24000}},
14 "output": map[string]any{"format": map[string]any{"type": "audio/pcm", "rate": 24000}, "voice": "alloy"},
15 },
16 },
17})
18
19// Stream microphone audio
20go func() {
21 for chunk := range micCh {
22 session.SendAudio(ctx, chunk)
23 }
24 session.Send(ctx, map[string]any{"type": "input_audio_buffer.commit"})
25 session.Send(ctx, map[string]any{"type": "response.create"})
26}()
27
28// Play response audio
29msgCh, _ := session.Events(ctx)
30for msg := range msgCh {
31 if msg.Audio != nil {
32 speaker.Write(msg.Audio)
33 }
34}

Error handling

Server errors arrive as a *meshapi.RealtimeError from Receive or on the errCh returned by Events:

1msg, err := session.Receive(ctx)
2if err != nil {
3 var re *meshapi.RealtimeError
4 if errors.As(err, &re) {
5 fmt.Println("code:", re.Code) // "invalid_api_key", "insufficient_quota", …
6 fmt.Println("message:", re.Message)
7 }
8}

Supported models

Model IDMode
openai/gpt-realtime-2Speech-to-speech
openai/gpt-realtime-1.5Speech-to-speech
openai/gpt-realtime-miniSpeech-to-speech
elevenlabs/scribe_v2_realtimeRealtime speech-to-text

Consult GET /v1/models for the current set of realtime-capable models.