Voice agents bring natural conversation to everyday workflows by coordinating three distinct workloads: speech recognition processes continuous audio, a language model reasons and calls tools, and speech synthesis generates audio that must keep pace with playback.
Drawing on our work at Baseten, this talk traces the evolution from optimized speech endpoints to a composable voice-agent stack. We’ll examine how each stage’s streaming, state, batching, and compute requirements shape infrastructure decisions, including model sizing, co-location, and independent scaling.
Following a conversation through the pipeline, we’ll identify opportunities to overlap execution and uncover delays in endpointing, inference, networking, and buffering. Implementation examples and benchmark findings will illustrate streaming capacity limits and the importance of tail latency across repeated turns. We’ll connect component benchmarks with end-to-end evaluations to guide model selection and improve conversational quality, responsiveness, and cost.
Key takeaways
- Translate the distinct characteristics of Speech-to-Text (STT), LLM, and Text-to-Speech (TTS) into serving, hardware, and scaling decisions.
- Identify critical-path delays and opportunities for overlapping execution, while accounting for tail latency across a conversation.
- Combine component benchmarks, representative application data, and conversational evaluations to guide model selection and system improvements.
Speaker
Lei Pan
Software Engineer @Baseten
Lei Pan is a software engineer at Baseten, where he builds infrastructure for voice AI and multimodal inference. His work focuses on model serving, distributed systems, and performance optimization, with an emphasis on latency, throughput, and reliability. Previously, he worked on GenAI infrastructure at Pinterest, including serving vision-language models for conversational applications. His experience spans optimizing model runtimes and building large-scale real-time GenAI products.
