Realtime-Bench reveals low-latency limits of current LLMs
AI
Reported by Dr. Kai Wei, Frontier AI Desk
- Realtime-Bench tests 50 interactive tasks to gauge temporal awareness, adaptive deliberation, and time-aware action selection in LLMs.
- GPT-6 Astra tops the wall-clock condition with a 36% pass rate, far below paused-clock performance of 70-80%.
- The benchmark highlights that current models struggle with real-time applications such as ticket buying or online auctions.
Researchers introduced Realtime-Bench, a 50-task suite measuring how well leading foundation models handle real-time decisions, exposing sub-70% success on timed challenges.
«Import AI 479: Spacewar; low-latency tasks versus LLMs; Genie 4»
Summary and commentary by Aura-V. Facts, figures and opinions belong to the original.
Researchers from the University of Texas at Austin, Ohio State, UIUC, NVIDIA, Northwestern and Steadyworks released Realtime-Bench, a suite of 50 dynamic tasks—30 web-based (travel, e-commerce, service, security) and 20 games (arcade, puzzle, strategy, simulation)—to probe AI agents’ low-latency capabilities. The benchmark measures temporal state awareness (RT-Perceive), adaptive deliberation (RT-Deliberate) and time-aware action selection (RT-Act) across three clock settings: wall clock, paused clock and token clock. The design forces models to track changing environments, limit reasoning as deadlines approach, and time actions appropriately, as illustrated by a ticket-purchase scenario.
The authors evaluated a range of models, including OpenAI’s GPT-6-Astra, GPT-5.6 Sol, GPT-Realtime-2, OpenAI Decisions, Anthropic’s Claude Fable 5.1 and Claude Opus 5, Google’s Gemini 3.8 Flash, Gemini 3.1 Flash Live, Gemini 3.8 Live, Jev 1.13, Deepseek V4.1 and Moonshot AI’s Kimi-K3. GPT-6 Astra achieved the highest wall-clock pass rate at 36%, while paused-clock scores rose to 70-80% for high-reasoning models, underscoring the latency gap.
Our read: The gap between paused-clock and wall-clock performance suggests that inference speed, not just model size, remains a critical bottleneck for deploying LLMs in real-time B2B workflows.
Interested in implementing this technology inside your company?
Our studio designs custom agent meshes, data pipelines, and outbound GTM systems for forward-looking enterprises.
