Speech to speech

Speech-to-speech models listen to the user and answer in their own voice, in one WebSocket session. AllModels serves them on two OpenAI-compatible endpoints; authenticate with Authorization: Bearer $ALLMODELS_API_KEY.

Choose an endpoint

Realtime is turn-based, and your app drives it. The user speaks, the turn ends (by the provider's voice detection or your commit), and the model replies as a response. Your app can trigger and cancel replies, trim what the user actually heard, and answer function calls. Use it when you need control: push-to-talk, scripted flows, tool-heavy agents.

Live is a continuous conversation, and the model drives it. Audio streams both ways all the time; the model decides when to speak, when to stop if the user talks over it, and when to hand a task to your app. Your app streams the microphone, plays the output, and answers those hand-offs. Use it for natural back-and-forth.

grok/grok-voice-2.0 and grok/grok-voice-think-fast-2.0 are deprecated by xAI and still served. Model ids work with or without the provider prefix.

Before you start

• Base URL. Set the OpenAI SDK baseURL to https://api.allmodels.io/oai; the SDK adds /realtime or /live/sessions.
• SDK. The examples were tested with openai 7.39 for JavaScript and 3.35 for Python. Live needs a version that includes LiveWS / client.live.connect().
• Audio. The examples send and receive 24 kHz, 16-bit, mono PCM. Convert a recording and play the reply with ffmpeg:

ffmpeg -i question.wav -f s16le -ac 1 -ar 24000 question.pcm
ffplay -f s16le -ar 24000 -ch_layout mono reply.pcm

Realtime

Use the provider's own Realtime events. The example sends question.pcm as one turn with turn detection off, then saves the spoken reply to reply.pcm.

import { once } from "node:events";
import { createWriteStream } from "node:fs";
import { readFile } from "node:fs/promises";
import OpenAI from "openai";
import { OpenAIRealtimeWS } from "openai/realtime/ws";

const client = new OpenAI({
  apiKey: process.env.ALLMODELS_API_KEY,
  baseURL: "https://api.allmodels.io/oai"
});
const realtime = new OpenAIRealtimeWS({ model: "openai/gpt-realtime-2.1-mini" }, client);
const reply = createWriteStream("reply.pcm");

realtime.on("response.output_audio.delta", ({ delta }) => {
  reply.write(Buffer.from(delta, "base64"));
});
realtime.on("response.output_audio_transcript.delta", ({ delta }) => {
  process.stdout.write(delta);
});
realtime.on("response.done", ({ response }) => {
  console.log("\nusage:", response.usage);
  reply.end();
  realtime.close();
});
realtime.on("error", console.error);

if (realtime.socket.readyState !== 1) await once(realtime.socket, "open");
const format = { type: "audio/pcm", rate: 24000 };
realtime.send({
  type: "session.update",
  session: {
    type: "realtime",
    output_modalities: ["audio"],
    audio: {
      input: { format, turn_detection: null },
      output: { format, voice: "marin" }
    }
  }
});

const audio = await readFile("question.pcm");
for (let offset = 0; offset < audio.length; offset += 4800) {
  realtime.send({
    type: "input_audio_buffer.append",
    audio: audio.subarray(offset, offset + 4800).toString("base64")
  });
}
realtime.send({ type: "input_audio_buffer.commit" });
realtime.send({ type: "response.create" });

Grok Voice

• Voices are xAI's (eve, ara, rex, …), not OpenAI's.
• A Grok session starts with turn detection off. Set turn_detection: { type: "server_vad" } if you want the model to reply when the user stops speaking.
• grok-voice-3.0 reasons by default; send session.reasoning.effort: "none" to turn that off.
• grok-voice-latest follows xAI's newest model (currently grok-voice-3.0). Pin grok-voice-3.0 to keep the same model.

Live

Send session.start as the first frame, stream the microphone with session.input_audio.append, and play session.output_audio.delta. To end, send session.close and wait for session.closed. Switch between gpt-live-1 and gemini-3.8-live by changing the model id and the voice: each model takes its own voice names.

import { createWriteStream } from "node:fs";
import { readFile } from "node:fs/promises";
import { setTimeout as sleep } from "node:timers/promises";
import OpenAI from "openai";
import { LiveWS } from "openai/resources/live/ws";

const client = new OpenAI({
  apiKey: process.env.ALLMODELS_API_KEY,
  baseURL: "https://api.allmodels.io/oai"
});
const live = new LiveWS(client);
const reply = createWriteStream("reply.pcm");
let quiet;

live.on("session.output_audio.delta", ({ delta }) => {
  reply.write(Buffer.from(delta, "base64"));
});
live.on("session.output_transcript.delta", ({ delta }) => {
  process.stdout.write(delta);
  // Close two seconds after the reply's transcript goes quiet.
  clearTimeout(quiet);
  quiet = setTimeout(() => live.send({ type: "session.close" }), 2000);
});
live.on("session.closed", ({ reason, usage }) => {
  console.log(`\nclosed: ${reason}, ${usage.seconds} s`);
  reply.end();
  live.close();
});
live.on("error", console.error);

live.send({
  type: "session.start",
  session: {
    model: "gpt-live-1", // or "gemini-3.8-live"
    instructions: "You are a helpful voice assistant. Keep replies short.",
    audio: {
      format: { type: "audio/pcm", rate: 24000 },
      output: { voice: "marin" } // for gemini-3.8-live, a Gemini voice such as "Zephyr"
    },
    delegation: { type: "client" }
  }
});

// Stream 100 ms frames in real time, then two seconds of silence,
// as a microphone would.
const question = await readFile("question.pcm");
const audio = Buffer.concat([question, Buffer.alloc(2 * 24000 * 2)]);
for (let offset = 0; offset < audio.length; offset += 4800) {
  live.send({
    type: "session.input_audio.append",
    audio: audio.subarray(offset, offset + 4800).toString("base64")
  });
  await sleep(100);
}

Delegation

When the model needs something it can't do itself (a calendar, an account, a search), it hands the task to your app with session.delegation.created. The event carries no task text: work out the request from the conversation so far (session.input_transcript.delta). Send progress with session.thinking.append, which the model reads but does not speak, and the result with session.commentary.append, which it speaks.

live.on("session.delegation.created", async ({ delegation }) => {
  live.send({ type: "session.thinking.append", delegation_id: delegation.id, content: "Checking the calendar." });
  const answer = await checkCalendar(); // your code
  live.send({ type: "session.commentary.append", delegation_id: delegation.id, content: answer });
});

While your app works, the model may tell the user it is checking. You can append commentary more than once on the same delegation, for example with a later update.

Session settings

Gemini on Live

• Gemini waits for the user to finish speaking before it answers; gpt-live-1 can start before they finish.
• A Gemini session ends after two hours, with session.closed reason expired.

Billing

Prices are on the models page.

If a session uses up your balance or hits a guardrail, it closes with 4402 or 4429. See WebSocket close codes.

Realtime and Live field names

Limits and errors

• Live first frame: a text session.start of at most 64 KB, within 15 seconds of connecting. Audio you send right after it is held until the session opens. A refused first frame gets an error event and the socket closes; nothing is billed.
• Wrong endpoint: a Live model on /oai/realtime returns 400 invalid_model.
• Ending a Live session: treat session.closed as the end; don't wait for the WebSocket close handshake.
• Discovery: GET /v1/models lists these models under speech_to_speech, each with the endpoints to connect to.

References: Realtime and Live connection parameters, and the WebSocket channels for Realtime and Live.