OpenAI Realtime Voice API: Getting Started Guide

Last verified: 2026-10-20

Updated October 20, 2026: OpenAI's Realtime guide now says: "If you still have a beta Realtime integration, migrate it to the GA interface before moving forward with new work." The WebSocket version of the Realtime Console that earlier versions of this guide walked through is built on the beta client (@openai/realtime-api-beta), so the setup, tool registration and audio samples below are rewritten for the GA interface, adapted from OpenAI's WebSocket guide and Realtime conversations guide. Per-token prices and how the API compares with hosted voice platforms are in our OpenAI Realtime directory entry.

Official Sources#

Most voice AI applications follow a three-step loop: record audio, send it to a transcription API, pass the text to an LLM, send the response to a text-to-speech API, play the audio back. Each hop adds latency. By the time the user hears a response, multiple round trips have happened.

OpenAI's Realtime API removes that overhead. It uses WebSockets to stream audio packets directly between your application and the model. As you speak, tiny audio chunks travel over the socket in real time. The moment you stop, the model already has the full payload and can begin responding. There is no transcription step, no separate TTS call. The model handles everything natively over a single persistent connection.

The result is voice interaction that feels conversational rather than transactional. And because the connection is stateful, the model remembers what was said earlier in the conversation without you manually managing chat history.

How It Works#

The key difference from the standard Chat Completions API is the transport layer. Instead of HTTP request/response pairs, the Realtime API maintains a WebSocket connection. Both sides can send messages at any time.

OpenAI's current guidance is that WebRTC is the better transport for browser and mobile clients, while WebSockets remain the right choice for server-to-server integrations where your backend already receives raw audio. The architecture below is the WebSocket path, which OpenAI still describes as "a great choice for connecting to the OpenAI Realtime API in server-to-server applications." The console's default version connects over WebRTC; its older WebSocket version, on the websockets branch, is built on the beta client, so this guide now follows OpenAI's GA WebSocket examples instead.

For the design side of the same problem, read OpenAI Codex: Cloud AI Coding With GPT-5.3 with OpenAI vs Anthropic in 2026 - Models, Tools, and Developer Experience; they show how agent-generated interfaces fail and how to give coding agents better visual constraints.

On the client side, your microphone captures audio and streams small packets to the server. On the server side, a relay process forwards those packets through the WebSocket to OpenAI. The model processes the audio, generates a response, and streams audio packets back. Your application plays them as they arrive.

This architecture means:

• No manual transcription - the model receives raw audio directly
• No separate TTS - the model generates audio output natively
• Stateful conversations - the model tracks conversation history server-side
• Function calling - the model can invoke tools mid-conversation, just like Chat Completions

Setting Up a GA Session#

OpenAI's console app used to be the starting point, but its WebSocket version (the websockets branch) imports @openai/realtime-api-beta, which makes it a beta integration. Per OpenAI's beta to GA migration notes, the GA interface drops the OpenAI-Beta: realtime=v1 header, requires session.type in session configuration, moves output audio settings under session.audio.output, and renames response events to names like response.output_audio.delta and response.output_text.delta.

On a server, the GA path needs only the ws package (npm install ws) and a standard API key that never leaves your backend. This is OpenAI's Node.js connection example from the WebSocket guide, with a session.update that sets the audio formats, turn detection and voice, adapted from the Realtime conversations guide:

import WebSocket from "ws";

const url = "wss://api.openai.com/v1/realtime?model=gpt-realtime-2.1";
const ws = new WebSocket(url, {
  headers: {
    Authorization: "Bearer " + process.env.OPENAI_API_KEY,
  },
});

ws.on("open", function open() {
  console.log("Connected to server.");

  ws.send(
    JSON.stringify({
      type: "session.update",
      session: {
        type: "realtime",
        instructions: "You are a helpful voice assistant.",
        output_modalities: ["audio"],
        audio: {
          input: {
            format: { type: "audio/pcm", rate: 24000 },
            turn_detection: { type: "semantic_vad" },
          },
          output: {
            format: { type: "audio/pcm", rate: 24000 },
            voice: "marin",
          },
        },
      },
    })
  );
});

ws.on("message", function incoming(message) {
  console.log(JSON.parse(message.toString()));
});

The server's first event is session.created. If your app identifies individual end users, OpenAI recommends also sending a stable, hashed user ID in the OpenAI-Safety-Identifier header on this connection.

For a browser or mobile client, OpenAI's getting-started guide now starts from the Agents SDK over WebRTC instead. Your server creates an ephemeral client secret with POST /v1/realtime/client_secrets, and the browser connects with it. From the Realtime guide:

import { RealtimeAgent, RealtimeSession } from "@openai/agents/realtime";

const agent = new RealtimeAgent({
  name: "Assistant",
  instructions: "You are a helpful voice assistant.",
});

const session = new RealtimeSession(agent, {
  model: "gpt-realtime-2.1",
});

await session.connect({
  apiKey: "ek_...(ephemeral key from your server)",
});

The rest of this guide stays on the WebSocket path, where your backend owns the socket.

Why a Relay Server?#

The relay server sits between your frontend and OpenAI's WebSocket endpoint. You need it because:

1. API key security - your OpenAI key stays on the server, never exposed to the browser
2. WebSocket proxying - the relay forwards audio packets between the browser WebSocket and the OpenAI WebSocket
3. Production path - in a deployed app, this is where you would add authentication, rate limiting, and usage tracking

In production, you would deploy the relay as a Node.js service behind your API. If the relay exists only to hide the key from a browser, the ephemeral client secret flow above is the alternative: your server mints a short-lived credential and the browser talks to OpenAI over WebRTC.

Function Calling Over Voice#

The most powerful feature of the Realtime API is function calling. You can define tools the same way you would with Chat Completions, and the model will invoke them mid-conversation based on what the user says.

Here are two example tools, a memory tool and a weather lookup, in the flat GA tool shape (type, name, description, parameters):

// Tool definitions, registered with session.update below
const tools = [
  {
    type: "function",
    name: "set_memory",
    description: "Saves important information that the user wants to remember.",
    parameters: {
      type: "object",
      properties: {
        key: {
          type: "string",
          description: "The label for this memory",
        },
        value: {
          type: "string",
          description: "The content to remember",
        },
      },
      required: ["key", "value"],
    },
  },
  {
    type: "function",
    name: "get_weather",
    description: "Gets the current weather for a given location.",
    parameters: {
      type: "object",
      properties: {
        location: {
          type: "string",
          description: "City name or location",
        },
      },
      required: ["location"],
    },
  },
];

// Register the tools for the whole session
ws.send(
  JSON.stringify({
    type: "session.update",
    session: { type: "realtime", tools, tool_choice: "auto" },
  })
);

Tools can be set for the session with session.tools in a session.update event, as above, or for a single response with response.tools in response.create.

When you say "What's the weather in Toronto?", the model recognizes this matches the get_weather tool, extracts the location parameter, and invokes the function. Your code fetches the weather data and sends the result back through the WebSocket. The model then speaks the answer.

The set_memory tool demonstrates persistent state. Say "Remember that I need to buy eggs tomorrow" and the model calls set_memory with the key and value. Your handler decides where the pair is stored, and the call and its result stay in the session's conversation for later turns.

Implementing a Tool Handler#

Here is how the weather function works under the hood:

async function handleGetWeather(location: string) {
  // Get coordinates from location name
  const geoResponse = await fetch(
    `https://geocoding-api.open-meteo.com/v1/search?name=${encodeURIComponent(location)}&count=1`
  );
  const geoData = await geoResponse.json();
  const { latitude, longitude } = geoData.results[0];

  // Get weather data
  const weatherResponse = await fetch(
    `https://api.open-meteo.com/v1/forecast?latitude=${latitude}&longitude=${longitude}&current_weather=true`
  );
  const weatherData = await weatherResponse.json();

  return {
    temperature: weatherData.current_weather.temperature,
    windSpeed: weatherData.current_weather.windspeed,
    unit: "celsius",
  };
}

To run it, watch for the model's function call and hand the result back. The completed call arrives in a response.done event as an output item with type: "function_call", a name, a JSON string of arguments and a call_id. You return the result as a function_call_output item with the same call_id, then send response.create so the model speaks the answer. Adapted from the function calling section of the conversations guide:

ws.on("message", async (message) => {
  const event = JSON.parse(message.toString());
  if (event.type !== "response.done") return;

  for (const item of event.response.output) {
    if (item.type !== "function_call" || item.name !== "get_weather") continue;

    const { location } = JSON.parse(item.arguments);
    const result = await handleGetWeather(location);

    ws.send(
      JSON.stringify({
        type: "conversation.item.create",
        item: {
          type: "function_call_output",
          call_id: item.call_id,
          output: JSON.stringify(result),
        },
      })
    );
    ws.send(JSON.stringify({ type: "response.create" }));
  }
});

The pattern is close to function calling in Chat Completions. Define the tool schema, handle the invocation, return structured data. The difference is the transport - everything happens as events over one socket instead of HTTP requests.

Stateful Conversations#

With the standard Chat Completions API, you manage conversation history yourself. Every request includes the full message array. The Realtime API handles this server-side. The model remembers everything said in the current session, and a session lasts at most 60 minutes, per the conversations guide.

This means you can have exchanges like:

"What's the weather in New York?" "The current temperature in New York is 17.4 degrees celsius..." "How about Chicago?"

The model understands "how about" refers to weather because it has the conversation context. No message array management on your side.

One thing to watch: sometimes function call responses arrive after the model has already started speaking. This is inherent to the WebSocket architecture - the model may begin generating a response before the tool result comes back. If the tool call is slow, you might get a "I'm unable to retrieve that right now" response, followed by the actual data. A second query will work because the data is now in context.

Building Your Own Tools#

The weather tool is a starting point. The real value is in the tools you add. Here is a complete example of a tool that fetches stock prices:

const stockTool = {
  type: "function",
  name: "get_stock_price",
  description: "Gets the current stock price for a given ticker symbol.",
  parameters: {
    type: "object",
    properties: {
      ticker: {
        type: "string",
        description: "Stock ticker symbol (e.g., AAPL, GOOGL, MSFT)",
      },
    },
    required: ["ticker"],
  },
};

async function handleGetStockPrice(ticker: string) {
  const response = await fetch(
    `https://api.example.com/stocks/${ticker}/price`
  );
  const data = await response.json();

  return {
    ticker: data.symbol,
    price: data.current_price,
    change: data.daily_change,
    currency: "USD",
  };
}

The tool definition tells the model what the function does and what arguments it needs. When a user says "What's the Apple stock price?", the model matches that to get_stock_price, extracts AAPL as the ticker, and invokes the function. Your code fetches the data, returns it, and the model speaks the result.

More ideas with high utility:

Calendar integration - "Schedule a meeting with Sarah tomorrow at 2pm" triggers a tool that creates a Google Calendar event.

Database queries - "How many users signed up this week?" invokes a tool that runs a SQL query and returns the count.

Smart home control - "Turn off the living room lights" sends a command to your home automation API.

Document retrieval - "What does our refund policy say?" searches a vector database and returns relevant passages.

Each tool follows the same pattern:

1. Define the function schema with a clear description and typed parameters
2. Register it in session.tools with a session.update event
3. Handle the invocation in your relay server
4. Return the result as a function_call_output item, then send response.create so the model can narrate it

You can register as many tools as you need. The model selects the right one based on the user's spoken request. If no tool matches, it responds with natural conversation instead.

Understanding the Audio Pipeline#

The WebSocket connection handles three types of messages:

Input audio - raw audio from the user's microphone, sent as base64 chunks in input_audio_buffer.append events as it is captured. In a browser, navigator.mediaDevices.getUserMedia() captures the microphone; the session configured above takes mono PCM16 at 24 kHz (audio/pcm, rate 24000).

Output audio - base64 audio chunks from the model in response.output_audio.delta events. The response.output_audio.done and response.done events carry no audio bytes, only transcriptions, so the client buffers the deltas for smooth playback.

Events - JSON messages for function calls, transcriptions, status updates, and conversation management. These are how tool calls get triggered and results get returned.

Over a WebSocket you own all three streams: buffer management, sample rate conversion and playback synchronization are your code. That is why OpenAI's conversations guide says: "To play output audio back on a client device like a web browser, we recommend using WebRTC rather than WebSockets". If you still capture in the browser and relay, the PCM16 conversion helpers below are adapted from the conversations guide. The capture uses ScriptProcessorNode for brevity; it is deprecated in the Web Audio API, and AudioWorklet is the modern replacement.

// Helpers adapted from OpenAI's Realtime conversations guide
function floatTo16BitPCM(float32Array: Float32Array): ArrayBuffer {
  const buffer = new ArrayBuffer(float32Array.length * 2);
  const view = new DataView(buffer);
  let offset = 0;
  for (let i = 0; i < float32Array.length; i++, offset += 2) {
    const s = Math.max(-1, Math.min(1, float32Array[i]));
    view.setInt16(offset, s < 0 ? s * 0x8000 : s * 0x7fff, true);
  }
  return buffer;
}

function base64EncodeAudio(float32Array: Float32Array): string {
  const bytes = new Uint8Array(floatTo16BitPCM(float32Array));
  let binary = "";
  const chunkSize = 0x8000; // 32KB chunk size
  for (let i = 0; i < bytes.length; i += chunkSize) {
    binary += String.fromCharCode(...bytes.subarray(i, i + chunkSize));
  }
  return btoa(binary);
}

// Browser capture at 24 kHz. `ws` is the socket to your relay, which
// forwards each event to OpenAI unchanged.
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
const audioContext = new AudioContext({ sampleRate: 24000 });
const source = audioContext.createMediaStreamSource(stream);
const processor = audioContext.createScriptProcessor(4096, 1, 1);

processor.onaudioprocess = (event) => {
  ws.send(
    JSON.stringify({
      type: "input_audio_buffer.append",
      audio: base64EncodeAudio(event.inputBuffer.getChannelData(0)),
    })
  );
};

source.connect(processor);
processor.connect(audioContext.destination);

The model detects speech boundaries automatically. Voice activity detection is on by default in speech-to-speech sessions, with server_vad as the default mode (the session above opts into semantic_vad). When you stop talking, it recognizes the end of your turn, commits the audio buffer, and begins generating a response. You do not need to implement voice activity detection yourself, and you can turn it off by setting turn_detection to null for push-to-talk, per the VAD guide.

Production Deployment#

The setup above runs locally with your API key in the relay server's environment. For production, you need:

• Authentication - add a login flow before allowing WebSocket connections. Without it, anyone can generate audio on your API key.
• Rate limiting - voice sessions are expensive. Limit concurrent connections and session duration.
• Audio handling - the samples above only stream audio back for playback. A production app might record conversations, generate transcripts, or route audio to other services.
• Error recovery - WebSocket connections drop. Implement reconnection logic with exponential backoff.

The relay server architecture already gives you the right separation. Your frontend connects to your relay. Your relay connects to OpenAI. Authentication and billing logic live in the relay layer.

What Makes This Different#

The Realtime API is not just "ChatGPT but with voice." The WebSocket transport fundamentally changes what is possible:

• Interrupt handling - you can start speaking while the model is responding, and with VAD on it cancels the response and listens. Over a WebSocket your client manages playback, so it must stop the audio and send conversation.item.truncate to drop the unplayed part
• Ambient listening - keep the connection open and the model can respond to ambient conversation
• Multi-modal output - the model can respond with both text and audio simultaneously
• Sub-second latency - because audio streams directly, there is no transcription or TTS bottleneck

For developers building voice assistants, customer support bots, accessibility tools, or any application where natural conversation matters, the Realtime API is one of the most capable options available right now. The WebSocket architecture means you can build interactions that feel like talking to a person rather than dictating commands to a machine.

Continue Reading#

• GPT-Live: OpenAI's Realtime Voice Architecture - how OpenAI rearchitected realtime voice for its next system, and what it means for this API's future
• Build a Voice Agent with ElevenLabs Agents - a competing speech-to-speech stack with its own agent runtime
• Text-to-Speech APIs for Developers in 2026 - where the Realtime API's built-in audio output sits against dedicated TTS providers
• Gemini 3.8 Live and 3.8 Live Extended Thinking - Google's audio-to-audio WebSocket alternative with per-minute audio pricing, benchmarked against this API's family
• How to Migrate Off whisper-1 and gpt-4o-transcribe - move to gpt-transcribe and gpt-live-transcribe before the February 26, 2027 shutdown
• OpenAI Realtime in the Voice Agents Directory - gpt-realtime-2.1 and GPT-Live prices with their source quotes
• Voice AI Agent Platforms Compared - the Realtime API beside hosted voice platforms, open source frameworks and other realtime APIs, every value quoted

Frequently Asked Questions#

What is the OpenAI Realtime API and how does it differ from Chat Completions?#

The Realtime API keeps a persistent WebRTC or WebSocket connection for bidirectional voice interaction instead of HTTP request/response pairs. OpenAI recommends WebRTC for browser and mobile clients and WebSockets for server-to-server integrations. Audio streams directly between your application and the model with no transcription or text-to-speech steps. The model processes raw audio input and generates audio output natively. The connection is stateful, so the model remembers conversation history server-side without you managing a message array.

Why do I need a relay server for the Realtime API?#

The relay server sits between your frontend and OpenAI's WebSocket endpoint for three reasons: your API key stays on the server and never gets exposed to the browser, the relay forwards audio packets between browser and OpenAI WebSockets, and it provides a production path for adding authentication, rate limiting, and usage tracking. For browser clients, the alternative is an ephemeral client secret from POST /v1/realtime/client_secrets plus a direct WebRTC connection.

Does the Realtime API support function calling?#

Yes. You define tools in a flat function shape and register them in session.tools with a session.update event. The model's call arrives as a function_call item in response.done; you return the result as a function_call_output item with the same call_id, then send response.create. The model selects the appropriate tool based on spoken requests and can invoke functions mid-conversation. If no tool matches, it responds with natural conversation instead.

How does the Realtime API handle conversation state?#

The API handles conversation history server-side. The model remembers everything said in the current session without you manually managing chat history. You can have exchanges where the model understands context from earlier turns, like asking "How about Chicago?" after a weather query about New York. The model knows "how about" refers to weather because it has the full conversation context.

What audio format does the Realtime API use?#

The session in this guide uses mono PCM16 at 24 kHz (audio/pcm, rate 24000) for input and output, set under session.audio. Input chunks go up as base64 in input_audio_buffer.append events, and output arrives as base64 in response.output_audio.delta events. Over a WebSocket your client handles buffering, sample rate conversion and playback, which is why OpenAI recommends WebRTC for playing audio on client devices.

What latency can I expect with the Realtime API?#

Sub-second latency is possible because audio streams directly over WebSockets. There is no transcription step and no separate TTS call. The model detects speech boundaries automatically and begins generating a response as soon as you stop speaking. You can also interrupt the model mid-response - start speaking while it's responding and it will stop to listen. Over a WebSocket, your client stops playback and truncates the interrupted response itself.

What are the main production considerations for deploying the Realtime API?#

Four areas need attention: authentication before allowing WebSocket connections so anyone cannot generate audio on your API key, rate limiting because voice sessions are expensive (limit concurrent connections and session duration), audio handling for recording conversations or generating transcripts, and error recovery with reconnection logic using exponential backoff when WebSocket connections drop.

Can function call responses arrive late in the Realtime API?#

Yes. Sometimes function call responses arrive after the model has already started speaking because the model may begin generating a response before the tool result comes back. If the tool call is slow, you might get "I'm unable to retrieve that right now" followed by the actual data. A second query will work because the data is then in context.