Text-based language agents have gotten very good at reasoning and tool calling, and that success carries over to voice — but only under ideal acoustic conditions. Once you put an agent into a realistic, full-duplex conversation, performance falls off a cliff. On a suite of 278 grounded tasks, GPT-5 with reasoning scores 85% pass@1 in text. The same tasks handed to state-of-the-art voice agents land at 31–51% under clean audio and 26–38% under realistic audio — roughly 30–45% of the underlying text capability surviving the jump to speech.
This talk introduces τ-voice, a benchmark built to measure that gap in the conditions agents actually ship into. We stress-test leading voice models against diverse accents, background noise, mid-turn interruptions, and telephony compression, then trace what those non-idealities do to multi-turn workflows. The failure mode is rarely a single bad transcription: small conversational distortions cascade into tool-calling errors and logical deviations several turns downstream. We also look at voice quality across different languages and what "good" actually means outside English.
The most useful finding is that the gap is actionable rather than merely diagnostic. 94.8% of agent failures trace back to domain-agnostic conversational primitives — turn-taking, repair, grounding, interruption handling — rather than to missing domain knowledge. That means the recoverable performance sits in a small, shared set of behaviors, and fixing them lifts every voice deployment rather than one vertical at a time. Attendees will leave with a concrete picture of where voice agents break, how to measure it, and which failures are worth engineering against first.
Key Takeaways:
- Voice agents retain only 30–45% of their text counterparts’ task capability under realistic conditions — a gap that clean-audio demos and text-based evals systematically hide.
- Acoustic non-idealities (accents, noise, interruptions, telephony compression) don’t just cause transcription errors; they cascade into tool-calling failures and logical deviations across multi-turn workflows.
- 94.8% of failures come from domain-agnostic conversational primitives, not domain knowledge — so the highest-leverage fixes are shared infrastructure, and teams should measure and invest there before adding vertical-specific tuning.
Speaker
Soham Ray
Research Engineer @Sierra, Previously Research Engineer at ASAPP and Conducted NLP Research at Cornell
Soham Ray is a Research Engineer at Sierra, where he works on voice and agents and leads work on τ-bench and τ-voice, benchmarks for evaluating agent capability in realistic conversational settings.
Before Sierra, he spent three and a half years as a Research Engineer at ASAPP, working with Prof. Kilian Weinberger on customer service AI automation and building production customer support agents. He previously conducted NLP research at Cornell with Prof. Claire Cardie on NLP-assisted automated theorem systems, and worked as a Data Scientist at Afiniti. He holds a CS degree from Cornell University and is based in the San Francisco Bay Area.
