One Infrastructure, Every Model: How Netflix Scales ML & GenAI

Building and scaling a machine learning inference platform that serves dozens of use cases- from homepage personalization to fraud detection- is as much an organizational challenge as a technical one. In this talk, we share lessons from a 5-year journey of consolidating Netflix's ML inference infrastructure into a single platform across recommendations, commerce, and more: what we got right, what surprised us, and what we're still figuring out. This Inference platform hosts models ranging from traditional ML to next-generation use cases powered by LLMs, all hosted in-house.

The scale of this system is massive - ~1 million inference requests/sec, 300+ models of various kinds, and rapid innovation to support several ongoing A/B experiments.

We'll cover the architectural decisions that made multi-tenancy practical (and the operational costs that came with them), how we support many latency regimes on a single platform, and why drawing the line between business logic and model logic is harder than it sounds. Most importantly, we'll talk about what consolidation unlocked for ML practitioners- a single deployment contract, one platform to innovate against, and dramatically less time spent coordinating across infrastructure teams and more time spent on the actual modeling problem.

Key Takeaways:

1. Consolidate or suffer: Supporting dozens of bespoke inference services inevitably results in redundant rebuilding of core infrastructure for every new model iteration.
2. Many latency regimes, one platform: From a few 10ms to over 300ms latency for real-time A/B experiments, determining the necessary operational mode must be a first-class architectural decision.
3. One platform, one contract: Providing ML practitioners a single platform to innovate against transforms model deployment from a complex infrastructure project into a focused modeling problem.
4. The blurring of business and model logic: Negotiating the boundary between the business application and the model remains a significant challenge; misalignment results in fragmented algorithmic logic across both layers.