Researchers from a multi-institution consortium have developed Realtime-Bench, a benchmark suite of dynamic tasks intended to test how well modern, general-purpose large language models (LLMs) perform when decisions must be made under real-time constraints. The benchmark focuses on scenarios where delayed actions directly affect the chance of success.
What the benchmark contains
Realtime-Bench includes 50 interactive tasks: 30 web tasks and 20 game tasks. Web tasks are grouped into four categories: travel, e-commerce, service, and security. Game tasks cover arcade, puzzle, strategy, and simulation genres.
The benchmark probes three specific capabilities:
- Temporal state awareness ("RT-Perceive"): the model’s ability to use observations to track changes in the environment.
- Adaptive deliberation ("RT-Deliberate"): the model’s capacity to adjust reasoning depth and duration according to available time.
- Time-aware action selection ("RT-Act"): the model’s ability to time its actions appropriately for the situation.
The authors give a ticket-purchase example: an agent must track sale status (RT-Perceive), limit further deliberation as the purchase window closes (RT-Deliberate), and decide whether to wait for a sale or buy while tickets remain (RT-Act).
Timing conditions used in the tests
The benchmark evaluates models under three timing regimes:
- Wall clock: measures performance with real-world latency to reflect actual network and inference delays.
- Paused clock: lets evaluators pause time arbitrarily to assess baseline capability without time pressure.
- Token clock: advances simulated time at a fixed rate per generated token; consuming more tokens delays action execution in simulated time regardless of actual inference latency.
Who built the benchmark
Realtime-Bench was developed by researchers at the University of Texas at Austin, Ohio State University, University of Illinois at Urbana-Champaign, NVIDIA, Northwestern University and Steadyworks.
Models tested and key results
The authors evaluated a range of frontier models, including OpenAI’s GPT-6-Astra, GPT-5.6 Sol, GPT-Realtime-2 and OpenAI Decisions; Anthropic’s Claude Fable 5.1 and Claude Opus 5; Google’s Gemini 3.8 Flash, Gemini 3.1 Flash Live and Gemini 3.8 Live; Jev 1.13; and Chinese models such as Deepseek V4.1 and Moonshot AI Kimi-K3.
Under the wall-clock condition, the best-performing model was OpenAI GPT-6 Astra (high), achieving a 36% pass rate. By contrast, when evaluated with a paused clock, higher-reasoning models reached roughly 70% or even 80% success rates.
Why this matters
Realtime-Bench highlights concrete settings in which current LLMs break down: tasks requiring tightly timed perception, adaptive deliberation, and action timing. The gap between paused-clock and wall-clock performance suggests that latency and time-aware behavior—not raw reasoning ability alone—limit LLMs’ reliability in real-time interactive applications such as online auctions, live games, or other low-latency services.
The authors have published a leaderboard and the full technical paper titled "Realtime-Bench: Can Frontier Models Solve Real-time Interactive Problems?" (arXiv), which provide more detailed results and methodology.
Conclusion
Realtime-Bench provides a practical, targeted tool to reveal where LLMs currently struggle in real-time interactions. The measured shortfalls advise caution when deploying autonomous agents in scenarios where latency and precise timing determine success.
