QCon AI New York 2025

Session

From OTEL to SLMs: Distilling Frontier Model Behaviour from Production Telemetry

December 16–17, 2025 · The New York Academy of Medicine

Wednesday Dec 17 / 01:20PM EST · Library Reading Room, 3rd Flr

Speaker

Ben O'Mahony

Principal AI Engineer @Thoughtworks, Startup Advisor, Speaker, Future Author and Time Magazine’s Person of the Year 2006

Full speaker profile

Abstract

Your production agents already generate rich training data: every interaction is captured in your OTEL traces. This talk shows how to distill that telemetry into datasets to fine-tune Small Language Models (SLMs) and turn the process into a repeatable platform capability. Learn how to extract, filter, and train on production data, then roll out distilled models safely with evaluation gates, canary releases, and continuous improvement cycles to build faster, cheaper, and self-improving AI systems grounded in your observability pipeline.