Building and scaling a machine learning inference platform that serves dozens of use cases- from homepage personalization to fraud detection- is as much an organizational challenge as a technical one. In this talk, we share lessons from a 5-year journey of consolidating Netflix's ML inference infrastructure into a single platform across recommendations, commerce, and more: what we got right, what surprised us, and what we're still figuring out. This Inference platform hosts models ranging from traditional ML to next-generation use cases powered by LLMs, all hosted in-house.
The scale of this system is massive - ~1 million inference requests/sec, 300+ models of various kinds, and rapid innovation to support several ongoing A/B experiments.
We'll cover the architectural decisions that made multi-tenancy practical (and the operational costs that came with them), how we support many latency regimes on a single platform, and why drawing the line between business logic and model logic is harder than it sounds. Most importantly, we'll talk about what consolidation unlocked for ML practitioners- a single deployment contract, one platform to innovate against, and dramatically less time spent coordinating across infrastructure teams and more time spent on the actual modeling problem.
Key Takeaways:
- Consolidate or suffer: Supporting dozens of bespoke inference services inevitably results in redundant rebuilding of core infrastructure for every new model iteration.
- Many latency regimes, one platform: From a few 10ms to over 300ms latency for real-time A/B experiments, determining the necessary operational mode must be a first-class architectural decision.
- One platform, one contract: Providing ML practitioners a single platform to innovate against transforms model deployment from a complex infrastructure project into a focused modeling problem.
- The blurring of business and model logic: Negotiating the boundary between the business application and the model remains a significant challenge; misalignment results in fragmented algorithmic logic across both layers.
Speaker
Rajat Shah
Staff Software Engineer @Netflix
Rajat is a Staff Software Engineer at Netflix, leading the technical architecture for the global ML Model Serving Infrastructure. Over a decade at Netflix and Amazon, he has specialized in building highly available distributed systems and stable, usable ML platforms that power recommender systems, search, and payments at scale. A recipient of Amazon’s prestigious "Just Do It" Award from Jeff Bezos for his bias for action, Rajat excels at abstracting the complexities of distributed computing to drive developer velocity and platform reliability. He holds a Master’s in Machine Learning from North Carolina State University, blending deep theoretical knowledge with a proven track record of solving massive-scale infrastructure challenges.
