From Signal To Impact: The Eval Loop Behind Doordash’s Agentic Ordering System

Running an LLM judge is the easy part of agent evaluation. The harder problem is building a system whose results are repeatable, diagnostic, and actionable. Production agents operate across multi-turn conversations, tool calls, specialized agents, and changing external state. An evaluation system has to account for all of this while producing a signal that engineers can trust and act on.

We faced this challenge while building DoorDash’s agentic ordering system. We needed to monitor quality in real user sessions, test changes before launch, and ensure that improvements measured offline would carry into production.

We built an evaluation harness around four components. Intent-aware rubrics translate our product-quality hypotheses into measurable criteria. A transcript builder turns raw traces into the evidence needed for each judgment. A conversation simulator and fixed tool responses make offline runs comparable. An LLM judge calibrated against human reviewers applies the same quality standard across offline and production sessions. The harness mirrors the agent architecture, evaluating routing, guardrails, and domain-specific capabilities where their failures can be understood most clearly.

Beyond measuring quality, the harness turns production signals into a continuous improvement loop. It helps us identify recurring problems, understand their causes, and validate changes before they reach users. This talk covers the architecture behind that feedback loop, the design decisions that made it trustworthy, and the lessons we learned using it to improve a production agent system.

Key Takeaways:

1. Design an evaluation harness for maximum signal and minimum noise. Use criterion-specific transcript views, eligibility-aware rubrics, and controlled tool fixtures so scores reflect agent behavior rather than irrelevant context or environmental drift.
2. Translate product-quality hypotheses into measurable criteria. Evaluate routing, guardrails, and domain capabilities where their failures occur, while allowing different user intents to define success differently.
3. Unify offline and online evaluation. Apply the same rubrics and calibrated judge to simulated and production sessions so pre-launch results remain connected to real user behavior.
4. Turn evaluation signals into a continuous improvement loop. Cluster recurring failures, connect them to implementation context, investigate likely causes, and validate proposed changes before rollout.