> ## Documentation Index
> Fetch the complete documentation index at: https://docs.inputapi.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Realtime text and audio integration

> Configure a GA session, send text or PCM audio, and save completed audio output.

Realtime upgrades `GET /v1/realtime?model=...` and uses session / conversation / response events. It differs from [Responses WebSocket](/en/guides/responses-websocket.md); changing only the URL is insufficient. This server-side example targets the GA protocol without legacy Beta headers or event names.

## Run text or audio output

Install dependencies and set the address, token, and a Realtime-capable model using [connection and runtime](/en/guides/websocket.md). By default the script sends a fixed text greeting. Text mode prints text; audio mode saves audio. Choose a supported voice using `API_VOICE`.

```sh theme={null}
export API_REALTIME_MODE="text"
node examples/realtime.mjs

export API_REALTIME_MODE="audio"
export API_VOICE="marin"
export API_AUDIO_OUTPUT="realtime-output.wav"
node examples/realtime.mjs
```

Each execution creates a separate session. An existing output file causes an immediate failure instead of overwriting; choose a new filename for another run. Output is a 24 kHz, mono, 16-bit little-endian PCM WAV file for a WAV-capable player. The example does not play sound directly.

## Send recorded speech

Set `API_AUDIO_PCM_PATH` to upload a file instead of the fixed text. It must contain **headerless PCM16LE, 24 kHz, mono**, not WAV, MP3, AAC, or Base64 text. Export that format using your recording/conversion tool. The example accepts 0.1–30 seconds and checks length, even byte count, and common container headers. Raw PCM alone cannot prove its real sample rate or speech quality.

```sh theme={null}
export API_AUDIO_PCM_PATH="/absolute/path/input.pcm"
export API_REALTIME_MODE="audio"
export API_AUDIO_OUTPUT="realtime-reply.wav"
node examples/realtime.mjs
```

Upload uses small Base64 chunks. The example disables VAD, appends and commits audio, waits for committed, then creates a response. It does not combine automatic VAD generation with duplicate manual creation. Text mode can produce a text answer to spoken input, but does not guarantee a verbatim transcript.

## Complete script

`examples/realtime.mjs` imports `websocket-client.mjs` from the same directory.

```javascript theme={null}
import { existsSync, readFileSync, statSync, writeFileSync } from 'node:fs';
import { openWebSocket } from './websocket-client.mjs';

async function waitFor(session, type) {
  for (;;) {
    const event = await session.receive();
    if (event.type === type) return event;
    if (event.type === 'rate_limits.updated') continue;
    if (type === 'input_audio_buffer.committed' && ['conversation.item.added', 'conversation.item.done'].includes(event.type)) continue;
    throw new Error('Unexpected session event.');
  }
}

let session;
try {
  const mode = process.env.API_REALTIME_MODE ?? 'text';
  const voice = process.env.API_VOICE ?? 'marin';
  const outputPath = process.env.API_AUDIO_OUTPUT ?? 'realtime-output.wav';
  const pcmPath = process.env.API_AUDIO_PCM_PATH;
  if (!['text', 'audio'].includes(mode) || !voice.trim()) throw new Error('Invalid Realtime mode or voice.');
  if (mode === 'audio' && existsSync(outputPath)) throw new Error('Output already exists.');
  let inputAudio;
  if (pcmPath) {
    const file = statSync(pcmPath);
    // Raw PCM16LE, mono, 24 kHz: this example accepts 0.1..30 seconds.
    if (!file.isFile() || file.size < 4800 || file.size > 1440000 || file.size % 2) throw new Error('Invalid PCM input size.');
    inputAudio = readFileSync(pcmPath);
    if (['RIFF', 'OggS', 'ID3'].some((tag) => inputAudio.subarray(0, tag.length).toString() === tag)) {
      throw new Error('Provide raw PCM, not a media container.');
    }
  }
  session = await openWebSocket('realtime', { model: process.env.API_MODEL ?? '' });
  const created = await waitFor(session, 'session.created');
  if (created.session?.type !== 'realtime' || !created.session.id) throw new Error('Invalid GA session.');
  const audio = { input: { format: { type: 'audio/pcm', rate: 24000 }, turn_detection: null } };
  if (mode === 'audio') audio.output = { format: { type: 'audio/pcm', rate: 24000 }, voice };
  await session.send({ type: 'session.update', session: {
    type: 'realtime', output_modalities: [mode], audio, tools: [],
    instructions: 'Answer briefly. If given audio, respond to what the speaker says.',
  } });
  const updated = (await waitFor(session, 'session.updated')).session;
  if (updated?.id !== created.session.id || updated.type !== 'realtime'
      || JSON.stringify(updated.output_modalities) !== JSON.stringify([mode])
      || updated.audio?.input?.turn_detection !== null
      || updated.audio.input.format?.type !== 'audio/pcm' || updated.audio.input.format.rate !== 24000) {
    throw new Error('Session configuration was not applied.');
  }
  if (mode === 'audio' && (updated.audio?.output?.format?.type !== 'audio/pcm'
      || updated.audio.output.format.rate !== 24000 || updated.audio.output.voice !== voice)) {
    throw new Error('Unexpected output audio configuration.');
  }
  if (inputAudio) {
    for (let offset = 0; offset < inputAudio.length; offset += 24000) {
      await session.send({ type: 'input_audio_buffer.append', audio: inputAudio.subarray(offset, offset + 24000).toString('base64') });
    }
    await session.send({ type: 'input_audio_buffer.commit' });
    const committed = await waitFor(session, 'input_audio_buffer.committed');
    if (typeof committed.item_id !== 'string' || !committed.item_id) throw new Error('Audio was not committed.');
  } else {
    await session.send({ type: 'conversation.item.create', item: {
      type: 'message', role: 'user', content: [{ type: 'input_text', text: 'Say hello in one short sentence.' }],
    } });
  }
  await session.send({ type: 'response.create', response: { output_modalities: [mode] } });
  let responseId;
  let itemId;
  let contentDone = false;
  let text = '';
  let bytes = 0;
  const audioChunks = [];
  for (;;) {
    const event = await session.receive();
    if (event.type === 'response.created') {
      if (responseId || typeof event.response?.id !== 'string' || !event.response.id) throw new Error('Invalid response identity.');
      responseId = event.response.id;
      continue;
    }
    if (event.response_id && event.response_id !== responseId) throw new Error('Unexpected response identity.');
    if (['response.audio.delta', 'response.text.delta', 'response.audio.done', 'response.text.done'].includes(event.type)) {
      throw new Error('Legacy Realtime events are not supported by this GA example.');
    }
    if ([`response.output_${mode}.delta`, `response.output_${mode}.done`].includes(event.type)) {
      if (!responseId || event.response_id !== responseId || event.output_index !== 0 || event.content_index !== 0
          || typeof event.item_id !== 'string' || !event.item_id || contentDone) throw new Error('Invalid content event.');
      if (itemId && itemId !== event.item_id) throw new Error('Mixed output items.');
      itemId = event.item_id;
      if (event.type.endsWith('.done')) {
        if (mode === 'text' && event.text !== text) throw new Error('Text completion mismatch.');
        contentDone = true;
      } else {
        if (typeof event.delta !== 'string' || !event.delta.length) throw new Error('Invalid content delta.');
        if (mode === 'text') {
          text += event.delta;
          bytes += Buffer.byteLength(event.delta);
        } else {
          const chunk = Buffer.from(event.delta, 'base64');
          if (!chunk.length || chunk.toString('base64') !== event.delta) throw new Error('Invalid Base64 audio.');
          audioChunks.push(chunk);
          bytes += chunk.length;
        }
        if (bytes > 10 * 1024 * 1024) throw new Error('Output limit exceeded.');
      }
    } else if (event.type === 'response.done') {
      const result = event.response;
      if (!responseId || result?.id !== responseId || result.status !== 'completed' || result.error || !contentDone
          || !Array.isArray(result.output) || result.output.length !== 1) throw new Error('Response did not complete.');
      const item = result.output[0];
      if (item.id !== itemId || item.type !== 'message' || item.role !== 'assistant' || item.status !== 'completed'
          || !Array.isArray(item.content) || item.content.length !== 1 || item.content[0].type !== `output_${mode}`) {
        throw new Error('Unexpected output, refusal, or tool call.');
      }
      if (mode === 'text') {
        if (!text.trim() || item.content[0].text !== text) throw new Error('Final text mismatch.');
        console.log(text);
      } else {
        if (!bytes || bytes % 2) throw new Error('Incomplete PCM sample data.');
        const header = Buffer.alloc(44);
        header.write('RIFF', 0); header.writeUInt32LE(36 + bytes, 4); header.write('WAVEfmt ', 8);
        header.writeUInt32LE(16, 16); header.writeUInt16LE(1, 20); header.writeUInt16LE(1, 22);
        header.writeUInt32LE(24000, 24); header.writeUInt32LE(48000, 28);
        header.writeUInt16LE(2, 32); header.writeUInt16LE(16, 34);
        header.write('data', 36); header.writeUInt32LE(bytes, 40);
        writeFileSync(outputPath, Buffer.concat([header, ...audioChunks]), { flag: 'wx' });
        console.log(`Saved ${bytes} PCM bytes to ${outputPath}`);
      }
      break;
    } else if (['response.failed', 'response.incomplete', 'response.cancelled'].includes(event.type)) {
      throw new Error('Response ended without success.');
    }
  }
} catch {
  console.error('Realtime failed. Check configuration, session, media format, events, and deadline; no automatic retry.');
  process.exitCode = 1;
} finally {
  session?.close();
}
```

## Session and audio validation

The script waits for `session.created`, sends `session.update`, and checks the effective modality, PCM format, sample rate, voice, and VAD settings in `session.updated`. It uses GA `response.output_text.*` / `response.output_audio.*` events; legacy `response.text.*` / `response.audio.*` events are incompatible. See the [Realtime conversation guide](https://developers.openai.com/api/docs/guides/realtime-conversations) for configuration.

The receiver permits one assistant output item with one content part and checks response\_id / item\_id / output\_index / content\_index. Text must match deltas, text.done, and final output. Audio bytes come from deltas, not the final response.done object. Publication requires a content done event, response.done with completed status, and valid nonempty output. See [Realtime server events](https://developers.openai.com/api/reference/resources/realtime/server-events).

Invalid Base64, odd total PCM length, missing done, failed / cancelled / incomplete states, refusals, tools, disconnects, and timeout fail the example. The 10 MiB output cap is an example bound. Individual audio chunks may split samples, but the joined length must be even. Fixtures check bytes and WAV headers, not speech intelligibility or real model availability.

## Multi-turn sessions, tools, and interruption

Production sessions can add conversation items and create subsequent responses, but applications must manage playback, event association, and context. This script generates one turn per execution. It does not capture microphones, implement automatic VAD conversations, execute tools, perform echo cancellation, or provide browser WebRTC.

Interruption requires distinguishing server generation cancellation from stopping local playback. Generated but unplayed audio may require `conversation.item.truncate` to synchronize heard history; upstream rules determine the fields. Do not transplant WebRTC audio-buffer clearing into WebSocket playback. See [Realtime client events](https://developers.openai.com/api/reference/resources/realtime/client-events).

Ordinary gateway forwarding does not automatically convert every legacy protocol into GA. WebRTC, SIP, temporary client tokens, and Live API need separate verification; this site does not promise those capabilities.

[Realtime API reference](/en/api-reference/realtime.md) · [Production integration](/en/guides/production.md)

[阅读中文](/zh/guides/realtime.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.