Building AI Agents for 127 Million Customers: Practical Lessons from Nubank

Scaling autonomous AI agents to serve over 127 million customers introduces significant challenges. This technical talk details the system design patterns and engineering principles employed at Nubank to build and operate production-grade, customer-facing AI agents. We will present our data-driven development lifecycle, which is centered on a comprehensive evaluation suite. We will cover the design of our automated evaluation pipelines, which quantitatively assess agent performance against key metrics and ensure consistent user experience at scale.

The discussion will then transition to the technical specifics of LLM prompt engineering, including strategies for structuring optimal prompts and generating precise tool schemas for complex, multi-turn actions. A primary focus will be on the implementation of feedback loops for automated system optimization. Attendees will learn how we utilize explicit and implicit feedback signals for the automated refinement of both evaluation datasets and agent system prompts. Finally, we will outline how these integrated systems create a flywheel for continuous improvement and share our online A/B testing frameworks and the key performance indicators (KPIs) used to measure in-production success and business impact.