Justin Uberti and Zahan Malkani, Members of Technical Staff at OpenAI, describe GPT‑Live, a third‑generation voice system designed to remove the latency and awkward turn taking of earlier voice AIs. Instead of relying on a separate turn detector, GPT‑Live uses a full‑duplex voice model that can listen and speak simultaneously, while delegating deeper reasoning and tool use asynchronously to frontier models such as GPT‑5.5.
Why turn‑based systems fell short
Earlier voice architectures inherited the turn‑based approach of text LLMs, representing each turn as a discrete audio blob. Cascaded pipelines—speech‑to‑text, LLM, text‑to‑speech—added latency and lost paralinguistic cues such as tone and pacing. Even speech‑to‑speech models that operate directly on audio still depended on a turn detector to decide when inference could start, so interaction remained segmented into turns.
GPT‑Live puts the voice model in control: audio streams in and out continuously through the model, while deeper thinking and tool calls occur off the live audio path. The system’s primary responsibility is to sustain an uninterrupted media loop; other work (invoking frontier models, persisting conversations) runs asynchronously.
System design for low latency
Over six months the team reworked inference, context management, and media transport to keep audio frames flowing on schedule. Key design choices include:
- Separating media flow from application/business logic so audio travels on a dedicated fast path, while delegation and backend work happen over an asynchronous RPC boundary.
- Rewriting the media frontend and inference logic in Go (replacing a Python asyncio implementation), which significantly improved frame delivery smoothness—the new system’s p95 latency matched the previous system’s p50.
- Using WebRTC as the transport foundation because it tolerates packet loss, clock drift, and client connection changes and can subtly stretch or accelerate audio to prevent gaps.
Minimizing buffering and blocking across the stack enables sub‑second responsiveness.
Managing stateful inference and handoffs
Long‑running voice sessions grow in context while model instances scale up and down. To handle that, GPT‑Live uses a seamless handoff mechanism:
- A replacement model instance is warmed alongside the existing one, prefilled with current session context.
- Inference is run in parallel on both instances and cutover occurs only when the new instance is fully ready.
The same mechanism supports dynamic context compaction: when a conversation’s context exceeds model limits, the system compacts context and prepares a replacement instance so the media loop never pauses. Heavy work like compaction is kept off the live path so conversations remain uninterrupted.
Asynchronous delegation to frontier models
GPT‑Live decouples "talking" from deeper "thinking" by letting the voice model keep the exchange flowing while delegating heavier reasoning or tool use to frontier models such as GPT‑5.5. To make this two‑model architecture feel seamless, the team optimized the entire delegation path:
- Prefilling frontier model inference sessions and required tools at voice session start so prompts are already processed before the first delegation.
- Maintaining stable session affinity and using prompt caching to reduce latency.
- Tuning reasoning effort, output limits, and tool schemas to minimize the time until a useful result is returned.
The voice model can briefly sustain the conversation while awaiting a delegated result, but the full delegation loop—routing, prompt processing, inference, and tool calls—is treated as part of the responsiveness budget.
Turning continuous audio into discrete messages for the rest of the product
Although the voice model processes continuous speech, many surrounding systems (ChatGPT’s UI, analytics, safety) still expect discrete user and assistant turns. The application server therefore:
- Uses partial transcripts and timing signals to infer who has the floor and builds a queue of provisional messages.
- Keeps the newest message provisional until attribution is reliable; speaker overlap and brief assistant acknowledgements are handled to avoid unnecessary fragmentation.
- Maintains two views of the conversation: a speculative view for the UI that can accept updates, and an authoritative final transcript for analytics.
This provides stable, turn‑based records for other systems without imposing turn taking on the live voice path.
Transport optimizations: WARP and Instant Connect
Startup overhead is critical because responsiveness must begin as soon as the user initiates a session. Traditional WebRTC session setup involves multiple network round trips, so the team introduced two optimizations:
- WARP (WebRTC Abridged Roundtrip Protocol): a set of backward‑compatible improvements that reduce media and data startup from six network round trips to one by piggybacking DTLS over ICE (SPED), using DTLS 1.3, pre‑negotiating SCTP (SNAP), and pre‑negotiating data channels instead of DCEP. WARP is published as open specifications and is being advanced in the IETF TSVWG working group; support has been added to libwebrtc and Pion.
- Instant Connect: pre‑negotiates SDP parameters outside the critical path so that, if those parameters are still valid, the server can materialize the session the moment the first media packet arrives. If pre‑negotiated parameters are stale, the normal signaling flow proceeds without extra latency.
Together, these allow a session to be started with a single UDP packet in many cases.
Production tests and operational lessons
Before routing real users, the team ran a silent test that mirrored a growing portion of ChatGPT Voice sessions to both the existing Advanced Voice Mode and the new system in shadow mode. Findings included:
- Capacity cannot be reduced to GPU throughput alone; continuous sessions require CPU‑side stream handlers, queues and network paths to scale as well.
- Geographic routing matters: sending sessions far from capacity increases latency. Co‑validating model rollouts with regional capacity and traffic‑steering improved results.
- Long sessions revealed memory, persistence, compaction, and shutdown race issues that short load tests did not expose.
- The team improved observability, added more granular telemetry, validated configurations against known‑good baselines, staged rollouts, and implemented controls to isolate or disable individual paths quickly.
The silent test therefore validated both technical capacity and operational readiness for failure detection and recovery.
Platform and future use
GPT‑Live’s architecture—streaming inference, a dedicated media path, asynchronous delegation and optimized transport—is already the foundation for broader realtime interactions. It powers ChatGPT Voice features (including newly launched desktop capabilities like controlling the computer and coordinating agents) and will underpin the forthcoming GPT‑Live API. Over time the platform aims to let voice experiences span more devices, apps and modalities while preserving the immediacy of live conversation.
If solving these engineering problems appeals to you, the authors invite interested engineers to join their team.
Authors: Justin Uberti and Zahan Malkani, Members of Technical Staff (OpenAI) — based on the original post.
