Realtime sessions
Run a voice session on the AllModels realtime protocol, with a speech-to-speech model or your own transcription, application and speech pipeline.
/v1/realtimeAllModelsA realtime session is one WebSocket that carries the user's audio in and the reply's audio out, using AllModels' own realtime.* events. A saved profile decides what runs behind it:
Profile type | What runs | Use it when |
|---|---|---|
speech_to_speech | One speech-to-speech model listens and answers in its own voice. | You want the model to handle the whole conversation. |
pipeline | Streaming transcription, then your application's text reply, then streaming speech. | Your own backend decides what to say. |
Both types use the same events, client secrets and SDK code. To use a provider's own protocol instead, see Speech to speech.
Create a speech-to-speech profile
curl https://api.allmodels.io/v1/realtime/profiles \
-H "Authorization: Bearer $ALLMODELS_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "support-voice",
"type": "speech_to_speech",
"speech_to_speech": {
"model": "openai/gpt-realtime-2.1-mini",
"voice": "marin",
"instructions": "You are a friendly support agent. Keep answers short.",
"turn_detection": { "type": "server_vad" }
}
}'The response's id (rtp_…) is the profile to connect with.
| Field | Values |
|---|---|
speech_to_speech.model | Required. A model listed under speech_to_speech in GET /v1/models with a native endpoint: OpenAI Realtime, Grok Voice and ElevenLabs Realtime models. |
speech_to_speech.voice | The model's own voice names. |
speech_to_speech.instructions | System instructions, up to 16,384 characters. |
speech_to_speech.turn_detection | { "type": "server_vad" }, { "type": "semantic_vad" } or { "type": "none" } (your app commits each turn). ElevenLabs models support semantic_vad (default) and none only. |
speech_to_speech.input_audio_format, output_audio_format | pcm_24000 (default) or ulaw_8000. |
speech_to_speech.input_transcription | { "enabled": true, "model": "gpt-4o-mini-transcribe" } to receive the user's transcript on OpenAI models, billed as speech-to-text. Grok and ElevenLabs send it without a setting. |
conversation.max_duration_seconds | Session limit, default 600. |
client.audio_transport | binary (default) or base64 for reply audio. |
Connect
From a server, connect with the profile and your API key:
wss://api.allmodels.io/v1/realtime?profile_id=rtp_…
Authorization: Bearer $ALLMODELS_API_KEYFrom a browser or mobile app, create a single-use client secret on your server and hand the client its url, which carries the secret, so your API key never leaves your server:
curl https://api.allmodels.io/v1/realtime/client-secrets \
-H "Authorization: Bearer $ALLMODELS_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "profile_id": "rtp_…", "overrides": { "speech_to_speech": { "voice": "cedar" } } }'overrides change settings for this session only; the saved profile is not changed.
Talk to the model
Send the microphone as binary frames of 16-bit mono PCM at the profile's input rate, and play the binary frames you receive. Audio you send right after connecting is held until the session is ready.
import WebSocket from "ws";
const ws = new WebSocket(`wss://api.allmodels.io/v1/realtime?profile_id=${profileId}`, {
headers: { Authorization: `Bearer ${process.env.ALLMODELS_API_KEY}` }
});
ws.on("message", (data, isBinary) => {
if (isBinary) return playAudio(data); // the reply's audio
const event = JSON.parse(data.toString());
if (event.type === "realtime.transcript.done") console.log("user:", event.transcript);
if (event.type === "realtime.response.text.delta") process.stdout.write(event.delta);
if (event.type === "realtime.response.done") console.log(`\nreply ${event.status}`);
if (event.type === "realtime.error") console.error(event.error.code, event.error.message);
});
ws.on("open", () => {
microphone.on("data", (pcm) => ws.send(pcm)); // 24 kHz, 16-bit, mono
});With turn_detection none, send realtime.input_audio.commit when the user finishes speaking. To send text instead of audio, send { "type": "realtime.input_text", "text": "…" }.
Interruptions
With voice activity detection on, the model stops when the user talks over it, and you receive realtime.response.interrupted. To stop a reply yourself, send realtime.response.cancel. Add played_ms, the milliseconds of the reply the listener actually heard, so the model remembers only what was played:
{ "type": "realtime.response.cancel", "played_ms": 1840 }Billing and limits
A speech-to-speech session is billed like the same model on /oai/realtime: see Speech to speech billing. If the balance runs out or a guardrail is hit, the session closes with 4402 or 4429 after a realtime.error.
Every event, field and close code: Realtime WebSocket reference.