AV-Dialog: Spoken Dialogue Models with Audio-Visual Input
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917080146640896 |
|---|---|
| author | Chen, Tuochao Veluri, Bandhav Gong, Hongyu Gollakota, Shyamnath |
| author_facet | Chen, Tuochao Veluri, Bandhav Gong, Hongyu Gollakota, Shyamnath |
| contents | Dialogue models falter in noisy, multi-speaker environments, often producing irrelevant responses and awkward turn-taking. We present AV-Dialog, the first multimodal dialog framework that uses both audio and visual cues to track the target speaker, predict turn-taking, and generate coherent responses. By combining acoustic tokenization with multi-task, multi-stage training on monadic, synthetic, and real audio-visual dialogue datasets, AV-Dialog achieves robust streaming transcription, semantically grounded turn-boundary detection and accurate responses, resulting in a natural conversational flow. Experiments show that AV-Dialog outperforms audio-only models under interference, reducing transcription errors, improving turn-taking prediction, and enhancing human-rated dialogue quality. These results highlight the power of seeing as well as hearing for speaker-aware interaction, paving the way for {spoken} dialogue agents that perform {robustly} in real-world, noisy environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_11124 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AV-Dialog: Spoken Dialogue Models with Audio-Visual Input Chen, Tuochao Veluri, Bandhav Gong, Hongyu Gollakota, Shyamnath Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Multimedia Sound Dialogue models falter in noisy, multi-speaker environments, often producing irrelevant responses and awkward turn-taking. We present AV-Dialog, the first multimodal dialog framework that uses both audio and visual cues to track the target speaker, predict turn-taking, and generate coherent responses. By combining acoustic tokenization with multi-task, multi-stage training on monadic, synthetic, and real audio-visual dialogue datasets, AV-Dialog achieves robust streaming transcription, semantically grounded turn-boundary detection and accurate responses, resulting in a natural conversational flow. Experiments show that AV-Dialog outperforms audio-only models under interference, reducing transcription errors, improving turn-taking prediction, and enhancing human-rated dialogue quality. These results highlight the power of seeing as well as hearing for speaker-aware interaction, paving the way for {spoken} dialogue agents that perform {robustly} in real-world, noisy environments. |
| title | AV-Dialog: Spoken Dialogue Models with Audio-Visual Input |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Multimedia Sound |
| url | https://arxiv.org/abs/2511.11124 |