Building Enterprise Realtime Voice Agents from Scratch: A Technical Tutorial
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917349500649472 |
|---|---|
| author | Qiu, Jielin Chen, Zixiang Yang, Liangwei Zhu, Ming Liu, Zhiwei Tan, Juntao Zhao, Wenting Murthy, Rithesh Ram, Roshan Prabhakar, Akshara Heinecke, Shelby Xiong, Caiming Savarese, Silvio Wang, Huan |
| author_facet | Qiu, Jielin Chen, Zixiang Yang, Liangwei Zhu, Ming Liu, Zhiwei Tan, Juntao Zhao, Wenting Murthy, Rithesh Ram, Roshan Prabhakar, Akshara Heinecke, Shelby Xiong, Caiming Savarese, Silvio Wang, Huan |
| contents | We present a technical tutorial for building enterprise-grade realtime voice agents from first principles. While end-to-end speech-to-speech models may ultimately provide the best latency for voice agents, fully self-hosted end-to-end solutions are not yet available. We evaluate the closest candidate, Qwen3-Omni, across three configurations: its cloud-only DashScope Realtime API achieves $\sim$702ms audio-to-audio latency with streaming, but is not self-hostable; its local vLLM deployment supports only the Thinker (text generation from audio, 516ms), not the Talker (audio synthesis); and its local Transformers deployment runs the full pipeline but at $\sim$146s -- far too slow for realtime. The cascaded streaming pipeline (STT $\rightarrow$ LLM $\rightarrow$ TTS) therefore remains the practical architecture for self-hosted realtime voice agents, and the focus of this tutorial. We build a complete voice agent using Deepgram (streaming STT), vLLM-served LLMs with function calling (streaming text generation), and ElevenLabs (streaming TTS), achieving a measured time-to-first-audio of 755ms (best case 729ms) with full function calling support. We release the full codebase as a 9-chapter progressive tutorial with working, tested code for every component. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_05413 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Building Enterprise Realtime Voice Agents from Scratch: A Technical Tutorial Qiu, Jielin Chen, Zixiang Yang, Liangwei Zhu, Ming Liu, Zhiwei Tan, Juntao Zhao, Wenting Murthy, Rithesh Ram, Roshan Prabhakar, Akshara Heinecke, Shelby Xiong, Caiming Savarese, Silvio Wang, Huan Sound We present a technical tutorial for building enterprise-grade realtime voice agents from first principles. While end-to-end speech-to-speech models may ultimately provide the best latency for voice agents, fully self-hosted end-to-end solutions are not yet available. We evaluate the closest candidate, Qwen3-Omni, across three configurations: its cloud-only DashScope Realtime API achieves $\sim$702ms audio-to-audio latency with streaming, but is not self-hostable; its local vLLM deployment supports only the Thinker (text generation from audio, 516ms), not the Talker (audio synthesis); and its local Transformers deployment runs the full pipeline but at $\sim$146s -- far too slow for realtime. The cascaded streaming pipeline (STT $\rightarrow$ LLM $\rightarrow$ TTS) therefore remains the practical architecture for self-hosted realtime voice agents, and the focus of this tutorial. We build a complete voice agent using Deepgram (streaming STT), vLLM-served LLMs with function calling (streaming text generation), and ElevenLabs (streaming TTS), achieving a measured time-to-first-audio of 755ms (best case 729ms) with full function calling support. We release the full codebase as a 9-chapter progressive tutorial with working, tested code for every component. |
| title | Building Enterprise Realtime Voice Agents from Scratch: A Technical Tutorial |
| topic | Sound |
| url | https://arxiv.org/abs/2603.05413 |