Speakers
Salina Wu
Software Engineer in ML Infrastructure @Pinterest - Building the Generative AI Serving Platform
Full speaker profile
Abstract
Pinterest has engineered a high-performance Vision-Language Model (VLM) serving stack built on NVIDIA Dynamo, designed to resolve the significant challenges of multimodal workloads, including heavy prefill computation and large KV cache pressure. By utilizing disaggregated inference, LMCache-based KV offloading, a multi-modal KV router, and custom PinCLIP projection embeddings—which map visual content directly into the model's token space—the architecture achieves 369x faster TTFT, 44x faster end-to-end latency, and 2.6x faster TPOT. To ensure scalability and reliability, the stack integrates several critical Dynamo components: Dynamo Planner, which provides LLM-optimized, SLA-based autoscaling; Dynamo Grove, which can leverage Volcano for consolidated, topology-aware Kubernetes scheduling; and the AI Configurator, an evaluation platform that simulates over 10,000 deployment configurations to predict latency and throughput, streamlining the transition from experimentation to optimized production deployment.
Main Takeaways:
Why VLM workloads are uniquely prefill-heavy and cache-sensitive
How KV-aware routing, E/PD disaggregation, and cache offloading improve efficiency at scale
Building a scalable, GPU-efficient K8s-based LLM serving platform
Building an LLM Evaluation Platform using Dynamo’s AI Configurator and DynoSim