Nubank cuts AI agent rollout time to 4.4 days using simulated customers
Dispatched by Tariq Al-Mansoor, Gulf & Capital Desk
- Nubank uses a simulator to vet AI agents before live deployment, shrinking rollout from 21.2 to 4.4 days.
- Live A/B tests show the new agent improves self-service rates by 4.9 points and transactional NPS by 36.7 points.
- An open-weight Qwen model replaced GPT-5.2, raising self-service by 8.8 points and cutting p95 latency by 25%.
Nubank’s AI team reduced the release cycle for its support agents from 21.2 days to 4.4 days by testing new versions with thousands of simulated customer conversations.
«How Nubank Ships AI Agents in Days, Not Weeks 🤖»
Summary and commentary by Aura-V. Facts, figures and opinions belong to the original.
Nubank announced that its AI team now validates each new support-agent version against thousands of simulated customers before any live rollout, cutting the average release time to 4.4 days from the previous 21.2 days Source. The approach, described in the “Screen Before You Serve” paper, lets an LLM judge score multi-turn dialogues and only promotes agents that meet a pre-set pass criterion to live A/B testing.
In the live test of Brazil’s highest-volume support agent, the new version scored 4.9 points higher on self-service rate and 36.7 points higher on transactional NPS than its predecessor. An open-weight Qwen model later replaced GPT-5.2 after screening across 29 configurations and 16,000 simulated conversations, delivering an additional 8.8-point lift in self-service, stable satisfaction, and a 25% reduction in p95 latency.
Our read: The bottleneck for AI-driven support now lies in rapid, reliable testing rather than model capability; Nubank’s simulator demonstrates a scalable method to accelerate iteration while maintaining performance safeguards.
Interested in implementing this technology inside your company?
Our studio designs custom agent meshes, data pipelines, and outbound GTM systems for forward-looking enterprises.