> ## Documentation Index
> Fetch the complete documentation index at: https://docs.inputapi.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Realtime 文字与语音接入

> 配置 GA 会话，发送文字或 PCM 音频并保存完成的音频响应。

Realtime 使用 `GET /v1/realtime?model=...` 升级连接，采用 session / conversation / response 事件。它与 [Responses WebSocket](/zh/guides/responses-websocket.md)不同，不能只替换 URL。本文提供服务端 GA 协议示例，不使用旧版 Beta 请求头或事件名。

## 运行文字或语音输出

按[连接与运行环境](/zh/guides/websocket.md)安装依赖、设置地址、令牌及支持 Realtime 的模型。默认发送固定的文字问候；text 模式输出文本，audio 模式保存音频。音色需模型支持，可替换 `API_VOICE`。

```sh theme={null}
export API_REALTIME_MODE="text"
node examples/realtime.mjs

export API_REALTIME_MODE="audio"
export API_VOICE="marin"
export API_AUDIO_OUTPUT="realtime-output.wav"
node examples/realtime.mjs
```

每次执行创建独立会话。音频文件已存在时立即退出，避免覆盖；再次执行请换输出文件名。输出为 24 kHz、单声道、16-bit little-endian PCM 的 WAV，可用支持 WAV 的播放器打开。示例没有直接扬声器播放。

## 发送已有语音

可设置 `API_AUDIO_PCM_PATH`，脚本改为上传文件而不是固定文字。文件必须是**无容器头的 PCM16LE、24 kHz、单声道**；不是 WAV、MP3、AAC 或 Base64 文本。请由录音/转码工具按此格式导出。示例允许 0.1–30 秒，检查长度、偶数字节和常见容器头，但无法仅从原始 PCM 判断实际采样率或语音质量。

```sh theme={null}
export API_AUDIO_PCM_PATH="/absolute/path/input.pcm"
export API_REALTIME_MODE="audio"
export API_AUDIO_OUTPUT="realtime-reply.wav"
node examples/realtime.mjs
```

上传按小块 Base64 编码。示例关闭 VAD，发送 append 后 commit，收到 committed 再 create；不会一边自动 VAD 生成、一边重复手动 create。可将模式设为 text 来获得对语音的文字回答，但这不是逐字转写保证。

## 完整脚本

`examples/realtime.mjs` 依赖同目录 `websocket-client.mjs`。

```javascript theme={null}
import { existsSync, readFileSync, statSync, writeFileSync } from 'node:fs';
import { openWebSocket } from './websocket-client.mjs';

async function waitFor(session, type) {
  for (;;) {
    const event = await session.receive();
    if (event.type === type) return event;
    if (event.type === 'rate_limits.updated') continue;
    if (type === 'input_audio_buffer.committed' && ['conversation.item.added', 'conversation.item.done'].includes(event.type)) continue;
    throw new Error('Unexpected session event.');
  }
}

let session;
try {
  const mode = process.env.API_REALTIME_MODE ?? 'text';
  const voice = process.env.API_VOICE ?? 'marin';
  const outputPath = process.env.API_AUDIO_OUTPUT ?? 'realtime-output.wav';
  const pcmPath = process.env.API_AUDIO_PCM_PATH;
  if (!['text', 'audio'].includes(mode) || !voice.trim()) throw new Error('Invalid Realtime mode or voice.');
  if (mode === 'audio' && existsSync(outputPath)) throw new Error('Output already exists.');
  let inputAudio;
  if (pcmPath) {
    const file = statSync(pcmPath);
    // Raw PCM16LE, mono, 24 kHz: this example accepts 0.1..30 seconds.
    if (!file.isFile() || file.size < 4800 || file.size > 1440000 || file.size % 2) throw new Error('Invalid PCM input size.');
    inputAudio = readFileSync(pcmPath);
    if (['RIFF', 'OggS', 'ID3'].some((tag) => inputAudio.subarray(0, tag.length).toString() === tag)) {
      throw new Error('Provide raw PCM, not a media container.');
    }
  }
  session = await openWebSocket('realtime', { model: process.env.API_MODEL ?? '' });
  const created = await waitFor(session, 'session.created');
  if (created.session?.type !== 'realtime' || !created.session.id) throw new Error('Invalid GA session.');
  const audio = { input: { format: { type: 'audio/pcm', rate: 24000 }, turn_detection: null } };
  if (mode === 'audio') audio.output = { format: { type: 'audio/pcm', rate: 24000 }, voice };
  await session.send({ type: 'session.update', session: {
    type: 'realtime', output_modalities: [mode], audio, tools: [],
    instructions: 'Answer briefly. If given audio, respond to what the speaker says.',
  } });
  const updated = (await waitFor(session, 'session.updated')).session;
  if (updated?.id !== created.session.id || updated.type !== 'realtime'
      || JSON.stringify(updated.output_modalities) !== JSON.stringify([mode])
      || updated.audio?.input?.turn_detection !== null
      || updated.audio.input.format?.type !== 'audio/pcm' || updated.audio.input.format.rate !== 24000) {
    throw new Error('Session configuration was not applied.');
  }
  if (mode === 'audio' && (updated.audio?.output?.format?.type !== 'audio/pcm'
      || updated.audio.output.format.rate !== 24000 || updated.audio.output.voice !== voice)) {
    throw new Error('Unexpected output audio configuration.');
  }
  if (inputAudio) {
    for (let offset = 0; offset < inputAudio.length; offset += 24000) {
      await session.send({ type: 'input_audio_buffer.append', audio: inputAudio.subarray(offset, offset + 24000).toString('base64') });
    }
    await session.send({ type: 'input_audio_buffer.commit' });
    const committed = await waitFor(session, 'input_audio_buffer.committed');
    if (typeof committed.item_id !== 'string' || !committed.item_id) throw new Error('Audio was not committed.');
  } else {
    await session.send({ type: 'conversation.item.create', item: {
      type: 'message', role: 'user', content: [{ type: 'input_text', text: 'Say hello in one short sentence.' }],
    } });
  }
  await session.send({ type: 'response.create', response: { output_modalities: [mode] } });
  let responseId;
  let itemId;
  let contentDone = false;
  let text = '';
  let bytes = 0;
  const audioChunks = [];
  for (;;) {
    const event = await session.receive();
    if (event.type === 'response.created') {
      if (responseId || typeof event.response?.id !== 'string' || !event.response.id) throw new Error('Invalid response identity.');
      responseId = event.response.id;
      continue;
    }
    if (event.response_id && event.response_id !== responseId) throw new Error('Unexpected response identity.');
    if (['response.audio.delta', 'response.text.delta', 'response.audio.done', 'response.text.done'].includes(event.type)) {
      throw new Error('Legacy Realtime events are not supported by this GA example.');
    }
    if ([`response.output_${mode}.delta`, `response.output_${mode}.done`].includes(event.type)) {
      if (!responseId || event.response_id !== responseId || event.output_index !== 0 || event.content_index !== 0
          || typeof event.item_id !== 'string' || !event.item_id || contentDone) throw new Error('Invalid content event.');
      if (itemId && itemId !== event.item_id) throw new Error('Mixed output items.');
      itemId = event.item_id;
      if (event.type.endsWith('.done')) {
        if (mode === 'text' && event.text !== text) throw new Error('Text completion mismatch.');
        contentDone = true;
      } else {
        if (typeof event.delta !== 'string' || !event.delta.length) throw new Error('Invalid content delta.');
        if (mode === 'text') {
          text += event.delta;
          bytes += Buffer.byteLength(event.delta);
        } else {
          const chunk = Buffer.from(event.delta, 'base64');
          if (!chunk.length || chunk.toString('base64') !== event.delta) throw new Error('Invalid Base64 audio.');
          audioChunks.push(chunk);
          bytes += chunk.length;
        }
        if (bytes > 10 * 1024 * 1024) throw new Error('Output limit exceeded.');
      }
    } else if (event.type === 'response.done') {
      const result = event.response;
      if (!responseId || result?.id !== responseId || result.status !== 'completed' || result.error || !contentDone
          || !Array.isArray(result.output) || result.output.length !== 1) throw new Error('Response did not complete.');
      const item = result.output[0];
      if (item.id !== itemId || item.type !== 'message' || item.role !== 'assistant' || item.status !== 'completed'
          || !Array.isArray(item.content) || item.content.length !== 1 || item.content[0].type !== `output_${mode}`) {
        throw new Error('Unexpected output, refusal, or tool call.');
      }
      if (mode === 'text') {
        if (!text.trim() || item.content[0].text !== text) throw new Error('Final text mismatch.');
        console.log(text);
      } else {
        if (!bytes || bytes % 2) throw new Error('Incomplete PCM sample data.');
        const header = Buffer.alloc(44);
        header.write('RIFF', 0); header.writeUInt32LE(36 + bytes, 4); header.write('WAVEfmt ', 8);
        header.writeUInt32LE(16, 16); header.writeUInt16LE(1, 20); header.writeUInt16LE(1, 22);
        header.writeUInt32LE(24000, 24); header.writeUInt32LE(48000, 28);
        header.writeUInt16LE(2, 32); header.writeUInt16LE(16, 34);
        header.write('data', 36); header.writeUInt32LE(bytes, 40);
        writeFileSync(outputPath, Buffer.concat([header, ...audioChunks]), { flag: 'wx' });
        console.log(`Saved ${bytes} PCM bytes to ${outputPath}`);
      }
      break;
    } else if (['response.failed', 'response.incomplete', 'response.cancelled'].includes(event.type)) {
      throw new Error('Response ended without success.');
    }
  }
} catch {
  console.error('Realtime failed. Check configuration, session, media format, events, and deadline; no automatic retry.');
  process.exitCode = 1;
} finally {
  session?.close();
}
```

## 会话与音频校验

示例等待 `session.created`，发送 `session.update`，再核对 `session.updated` 实际生效的模态、PCM 格式、采样率、音色和 VAD 设置。生成使用 GA 的 `response.output_text.*` / `response.output_audio.*` 事件；旧 `response.text.*` / `response.audio.*` 不兼容此示例。协议配置依据 [Realtime 会话指南](https://developers.openai.com/api/docs/guides/realtime-conversations)。

接收器限定一个助手输出项及一个内容块，核对 response\_id / item\_id / output\_index / content\_index。文本还会比对 delta、text.done 和最终 output；音频从 delta 收集字节，不能从 response.done 取完整音频。只有内容 done、response.done 的 completed 状态、非空有效输出全部满足后才发布结果。事件格式参考 [Realtime server events](https://developers.openai.com/api/reference/resources/realtime/server-events)。

非法 Base64、奇数字节 PCM、缺失 done、failed / cancelled / incomplete、拒答、工具输出、断连和超时都会失败。输出限制 10 MiB 是示例边界。原始 PCM 片段可以切在样本中间，最终拼接长度必须为偶数。模拟测试只检查字节和 WAV 头，不证明真实语音可懂或模型会话可用。

## 多轮、工具和打断

生产会话可继续添加 conversation item 并创建下一轮响应，但需要自己管理输出播放、事件关联和上下文。本脚本每次只生成一轮，不实现麦克风采集、VAD 自动对话、工具执行、回声消除或浏览器 WebRTC。

打断时要区分服务器生成取消与本地播放停止；音频已经生成但未播放的部分可能需要用 `conversation.item.truncate` 同步已听历史，具体字段由上游协议决定。不要给 WebSocket 照搬 WebRTC 的音频缓冲清理流程。参考 [Realtime client events](https://developers.openai.com/api/reference/resources/realtime/client-events)。

项目普通转发不会把所有旧版协议自动转换为 GA。WebRTC、SIP、临时客户端令牌及 Live API 需独立核实；本站没有注册这些能力的承诺。

[Realtime API 说明](/zh/api-reference/realtime.md) · [生产接入](/zh/guides/production.md)

[Read in English](/en/guides/realtime.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.