Speaker
Abstract
As organizations adopt AI at scale, evaluation becomes the backbone of trust, safety, and product readiness. Yet building effective evals is deceptively hard: there is no single metric or benchmark that captures the complexity of user-facing AI systems. This talk unpacks the evolving landscape of AI evaluations, drawing lessons from personalization infrastructure and enterprise adoption.
Evals aren’t just “does the model answer right?”—they span from model correctness to infra robustness, product safety, human experience, and long-term systemic effects. We’ll explore the multi-layered nature of evals: from core metrics like precision and recall, to system-level robustness (latency, drift, stability), to product-facing guardrails that ensure fairness, safety, and alignment with user expectations. AI adoption in enterprises fails if evals stop at precision/recall and don’t extend to guardrails, human experience, and systemic impact. We’ll discuss why manual labeling and gold-standard datasets remain indispensable for grounding LLM judgments, and how iterative evaluation loops allow criteria to evolve alongside product goals, avoiding the trap of static benchmarks.
The session will also address organizational dimensions: who owns evaluation, how to align infra, product, and ML teams on rigor, and what “good” looks like when rolling AI systems into production responsibly. Attendees will gain a framework for understanding evaluation as a dynamic, multi-layered practice that spans technical correctness, human experience, and long-term product impact.
Key Takeaways:
- Evals are multi-layered, spanning model correctness, infra reliability, product guardrails, and user experience.
- Manual labeling and curated datasets are essential for grounding AI judgments.
- Iterative evaluation loops ensure relevance as products and models evolve.
- Ownership and governance of evals are as much organizational as technical.
- A structured, layered approach accelerates safe, impactful AI adoption.