From Speech Models to Voice Agents

Voice agents bring natural conversation to everyday workflows by coordinating three distinct workloads: speech recognition processes continuous audio, a language model reasons and calls tools, and speech synthesis generates audio that must keep pace with playback.

Drawing on our work at Baseten, this talk traces the evolution from optimized speech endpoints to a composable voice-agent stack. We’ll examine how each stage’s streaming, state, batching, and compute requirements shape infrastructure decisions, including model sizing, co-location, and independent scaling.

Following a conversation through the pipeline, we’ll identify opportunities to overlap execution and uncover delays in endpointing, inference, networking, and buffering. Implementation examples and benchmark findings will illustrate streaming capacity limits and the importance of tail latency across repeated turns. We’ll connect component benchmarks with end-to-end evaluations to guide model selection and improve conversational quality, responsiveness, and cost.

Key takeaways:

1. Translate the distinct characteristics of Speech-to-Text (STT), LLM, and Text-to-Speech (TTS) into serving, hardware, and scaling decisions.
2. Identify critical-path delays and opportunities for overlapping execution, while accounting for tail latency across a conversation.
3. Combine component benchmarks, representative application data, and conversational evaluations to guide model selection and system improvements.