OpenAI realtime

AllModels serves two OpenAI-compatible realtime endpoints:

Endpoint	Use it for
`/oai/realtime`	Realtime transcription (`intent=transcription` or a transcription model): audio in, transcripts out. Speech to speech when `model` is a speech-to-speech model: OpenAI Realtime, Grok Voice or ElevenLabs Realtime.
`/oai/live/sessions`	Voice conversation over the GPT-Live protocol (`gpt-live-1`, `gemini-3.8-live`). See OpenAI Live.

Realtime transcription

Use any streaming speech-to-text model in the catalog, from any provider, not only OpenAI's: for example deepgram/nova-3, soniox/stt-rt-v5, elevenlabs/scribe-v2-realtime or openai/gpt-realtime-whisper. File-only models such as openai/whisper-1 are not available here. GET /v1/models marks each model's providers with streaming: true when they support it.

Connect to wss://api.allmodels.io/oai/realtime?intent=transcription, configure the session, and send base64-encoded PCM audio. Commit the audio buffer when an utterance is complete. A transcription session returns transcripts only; to turn a transcript into streamed speech, see Stream generated speech.

SDK clients

Use the official OpenAI JavaScript or Python SDK.

import { once } from "node:events";
import { readFile } from "node:fs/promises";
import OpenAI from "openai";
import { OpenAIRealtimeWebSocket } from "openai/realtime/websocket";

const client = new OpenAI({
  apiKey: process.env.ALLMODELS_API_KEY,
  baseURL: "https://api.allmodels.io/oai"
});
const realtime = await OpenAIRealtimeWebSocket.create(client, {
  model: "openai/gpt-realtime-whisper"
});
if (realtime.socket.readyState !== 1) await once(realtime.socket, "open");

realtime.on("conversation.item.input_audio_transcription.delta", (event) => {
  process.stdout.write(event.delta);
});
realtime.on("conversation.item.input_audio_transcription.completed", (event) => {
  console.log("\nfinal:", event.transcript);
});
realtime.on("error", console.error);

realtime.send({
  type: "session.update",
  session: {
    type: "transcription",
    audio: {
      input: {
        format: { type: "audio/pcm", rate: 24000 },
        transcription: {
          model: "openai/gpt-realtime-whisper",
          language: "en"
        },
        turn_detection: null
      }
    }
  }
});

const pcm = await readFile("chunk.pcm");
realtime.send({
  type: "input_audio_buffer.append",
  audio: pcm.toString("base64")
});
realtime.send({ type: "input_audio_buffer.commit" });

Connection details

WSwss://api.allmodels.io/oai/realtime?intent=transcriptionOpenAI

The OpenAI SDK authenticates with the API key passed to OpenAI or AsyncOpenAI. Browser connections can pass realtime and openai-insecure-api-key.<key> as WebSocket subprotocols, but the SDK examples use a backend connection.

Browser credentials are visible

A key placed in a WebSocket subprotocol is available to the browser application and developer tools. Prefer a backend connection for long-lived tenant keys.

Events and buffer flow

Direction	Event	Purpose
Client → server	`session.update`	Configure model, language, audio format, turn detection, provider, and provider options.
Client → server	`input_audio_buffer.append`	Add one base64-encoded audio chunk.
Client → server	`input_audio_buffer.commit`	Finalize the current utterance.
Client → server	`input_audio_buffer.clear`	Drop buffered audio.
Server → client	`session.created` / `session.updated`	Confirm effective session configuration.
Server → client	`input_audio_buffer.committed`	Confirm a committed item and its ID.
Server → client	`conversation.item.input_audio_transcription.delta`	Partial transcript text.
Server → client	`conversation.item.input_audio_transcription.completed`	Final transcript and duration usage.
Server → client	`error`	Protocol, provider, billing, or connection failure.

Choose the provider, model, and audio format before sending audio. These settings cannot be changed after the first audio chunk.

Provider selection

Set provider preferences with session.update:

realtime.send({
  type: "session.update",
  session: {
    type: "transcription",
    provider: "deepgram",
    provider_options: {
      keyterms: ["AllModels"],
      smart_format: true
    },
    audio: {
      input: {
        format: { type: "audio/pcm", rate: 24000 },
        transcription: { model: "deepgram/nova-3", language: "en" },
        turn_detection: null
      }
    }
  }
});

Speech to speech

Connect to /oai/realtime with one of the speech-to-speech models below and use the provider's own Realtime events. Unlike transcription, voice conversation works only with these models. With the SDK, pass the model to realtime.connect() (Python) or OpenAIRealtimeWS (JavaScript); to switch provider, change the model id.

Provider	Models
OpenAI	`openai/gpt-realtime-2.1`, `openai/gpt-realtime-2.1-mini`, `openai/gpt-realtime-2`
Grok Voice	`grok/grok-voice-3.0`, `grok/grok-voice-latest`
ElevenLabs Realtime	`elevenlabs/eleven_realtime_v1_mini`, `elevenlabs/eleven_realtime_v1_max`

import { once } from "node:events";
import OpenAI from "openai";
import { OpenAIRealtimeWS } from "openai/realtime/ws";

const client = new OpenAI({
  apiKey: process.env.ALLMODELS_API_KEY,
  baseURL: "https://api.allmodels.io/oai"
});
// Or "grok/grok-voice-3.0", "elevenlabs/eleven_realtime_v1_mini", ...
const realtime = new OpenAIRealtimeWS({ model: "openai/gpt-realtime-2.1-mini" }, client);
if (realtime.socket.readyState !== 1) await once(realtime.socket, "open");

realtime.send({
  type: "session.update",
  session: {
    type: "realtime",
    output_modalities: ["audio"],
    audio: { output: { voice: "marin" } }
  }
});
// Stream input_audio_buffer.append, then play response.output_audio.delta.

For a complete example that sends a recorded question and saves the spoken reply, see Speech to speech.

Differences from OpenAI

Events pass through unchanged in both directions, in each provider's own Realtime protocol.
Voices are each provider's: OpenAI's for OpenAI models, xAI's for Grok Voice (eve, ara, rex, …), ElevenLabs' for ElevenLabs Realtime. See Speech to speech for each provider's notes.
Input transcription (session.audio.input.transcription.model) on an OpenAI model is billed separately as speech-to-text at that transcription model's price. A transcription model without a price is answered with an error event (model_not_priced) and the session.update is not applied; the session stays open.
Live models (gpt-live-1, gemini-3.8-live) are refused with 400 invalid_model; use OpenAI Live.

Event reference: client events and server events.

AllModels serves two OpenAI-compatible realtime endpoints:

Endpoint	Use it for
`/oai/realtime`	Realtime transcription (`intent=transcription` or a transcription model): audio in, transcripts out. Speech to speech when `model` is a speech-to-speech model: OpenAI Realtime, Grok Voice or ElevenLabs Realtime.
`/oai/live/sessions`	Voice conversation over the GPT-Live protocol (`gpt-live-1`, `gemini-3.8-live`). See OpenAI Live.

Realtime transcription

SDK clients

Use the official OpenAI JavaScript or Python SDK.

import { once } from "node:events";
import { readFile } from "node:fs/promises";
import OpenAI from "openai";
import { OpenAIRealtimeWebSocket } from "openai/realtime/websocket";

const client = new OpenAI({
  apiKey: process.env.ALLMODELS_API_KEY,
  baseURL: "https://api.allmodels.io/oai"
});
const realtime = await OpenAIRealtimeWebSocket.create(client, {
  model: "openai/gpt-realtime-whisper"
});
if (realtime.socket.readyState !== 1) await once(realtime.socket, "open");

realtime.on("conversation.item.input_audio_transcription.delta", (event) => {
  process.stdout.write(event.delta);
});
realtime.on("conversation.item.input_audio_transcription.completed", (event) => {
  console.log("\nfinal:", event.transcript);
});
realtime.on("error", console.error);

realtime.send({
  type: "session.update",
  session: {
    type: "transcription",
    audio: {
      input: {
        format: { type: "audio/pcm", rate: 24000 },
        transcription: {
          model: "openai/gpt-realtime-whisper",
          language: "en"
        },
        turn_detection: null
      }
    }
  }
});

const pcm = await readFile("chunk.pcm");
realtime.send({
  type: "input_audio_buffer.append",
  audio: pcm.toString("base64")
});
realtime.send({ type: "input_audio_buffer.commit" });

Connection details

WSwss://api.allmodels.io/oai/realtime?intent=transcriptionOpenAI

Browser credentials are visible

A key placed in a WebSocket subprotocol is available to the browser application and developer tools. Prefer a backend connection for long-lived tenant keys.

Events and buffer flow

Direction	Event	Purpose
Client → server	`session.update`	Configure model, language, audio format, turn detection, provider, and provider options.
Client → server	`input_audio_buffer.append`	Add one base64-encoded audio chunk.
Client → server	`input_audio_buffer.commit`	Finalize the current utterance.
Client → server	`input_audio_buffer.clear`	Drop buffered audio.
Server → client	`session.created` / `session.updated`	Confirm effective session configuration.
Server → client	`input_audio_buffer.committed`	Confirm a committed item and its ID.
Server → client	`conversation.item.input_audio_transcription.delta`	Partial transcript text.
Server → client	`conversation.item.input_audio_transcription.completed`	Final transcript and duration usage.
Server → client	`error`	Protocol, provider, billing, or connection failure.

Choose the provider, model, and audio format before sending audio. These settings cannot be changed after the first audio chunk.

Provider selection

Set provider preferences with session.update:

realtime.send({
  type: "session.update",
  session: {
    type: "transcription",
    provider: "deepgram",
    provider_options: {
      keyterms: ["AllModels"],
      smart_format: true
    },
    audio: {
      input: {
        format: { type: "audio/pcm", rate: 24000 },
        transcription: { model: "deepgram/nova-3", language: "en" },
        turn_detection: null
      }
    }
  }
});

Speech to speech

Provider	Models
OpenAI	`openai/gpt-realtime-2.1`, `openai/gpt-realtime-2.1-mini`, `openai/gpt-realtime-2`
Grok Voice	`grok/grok-voice-3.0`, `grok/grok-voice-latest`
ElevenLabs Realtime	`elevenlabs/eleven_realtime_v1_mini`, `elevenlabs/eleven_realtime_v1_max`

import { once } from "node:events";
import OpenAI from "openai";
import { OpenAIRealtimeWS } from "openai/realtime/ws";

const client = new OpenAI({
  apiKey: process.env.ALLMODELS_API_KEY,
  baseURL: "https://api.allmodels.io/oai"
});
// Or "grok/grok-voice-3.0", "elevenlabs/eleven_realtime_v1_mini", ...
const realtime = new OpenAIRealtimeWS({ model: "openai/gpt-realtime-2.1-mini" }, client);
if (realtime.socket.readyState !== 1) await once(realtime.socket, "open");

realtime.send({
  type: "session.update",
  session: {
    type: "realtime",
    output_modalities: ["audio"],
    audio: { output: { voice: "marin" } }
  }
});
// Stream input_audio_buffer.append, then play response.output_audio.delta.

For a complete example that sends a recorded question and saves the spoken reply, see Speech to speech.

Differences from OpenAI

Events pass through unchanged in both directions, in each provider's own Realtime protocol.
Voices are each provider's: OpenAI's for OpenAI models, xAI's for Grok Voice (eve, ara, rex, …), ElevenLabs' for ElevenLabs Realtime. See Speech to speech for each provider's notes.
Input transcription (session.audio.input.transcription.model) on an OpenAI model is billed separately as speech-to-text at that transcription model's price. A transcription model without a price is answered with an error event (model_not_priced) and the session.update is not applied; the session stays open.
Live models (gpt-live-1, gemini-3.8-live) are refused with 400 invalid_model; use OpenAI Live.

Event reference: client events and server events.

Realtime transcription

SDK clients

Connection details

Events and buffer flow

Provider selection

Speech to speech

Differences from OpenAI

On this page

OpenAI realtime

Realtime transcription

SDK clients

Connection details

Events and buffer flow

Provider selection

Speech to speech

Differences from OpenAI

On this page