TAVID: Text-Driven Audio-Visual Interactive Dialogue Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909974658023424 |
|---|---|
| author | Kim, Ji-Hoon Ahn, Junseok Kwak, Doyeop Chung, Joon Son Watanabe, Shinji |
| author_facet | Kim, Ji-Hoon Ahn, Junseok Kwak, Doyeop Chung, Joon Son Watanabe, Shinji |
| contents | The objective of this paper is to jointly synthesize interactive videos and conversational speech from text and reference images. With the ultimate goal of building human-like conversational systems, recent studies have explored talking or listening head generation as well as conversational speech generation. However, these works are typically studied in isolation, overlooking the multimodal nature of human conversation, which involves tightly coupled audio-visual interactions. In this paper, we introduce TAVID, a unified framework that generates both interactive faces and conversational speech in a synchronized manner. TAVID integrates face and speech generation pipelines through two cross-modal mappers (i.e., a motion mapper and a speaker mapper), which enable bidirectional exchange of complementary information between the audio and visual modalities. We evaluate our system across four dimensions: talking face realism, listening head responsiveness, dyadic interaction fluency, and speech quality. Extensive experiments demonstrate the effectiveness of our approach across all these aspects. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_20296 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TAVID: Text-Driven Audio-Visual Interactive Dialogue Generation Kim, Ji-Hoon Ahn, Junseok Kwak, Doyeop Chung, Joon Son Watanabe, Shinji Computer Vision and Pattern Recognition Artificial Intelligence Audio and Speech Processing Image and Video Processing The objective of this paper is to jointly synthesize interactive videos and conversational speech from text and reference images. With the ultimate goal of building human-like conversational systems, recent studies have explored talking or listening head generation as well as conversational speech generation. However, these works are typically studied in isolation, overlooking the multimodal nature of human conversation, which involves tightly coupled audio-visual interactions. In this paper, we introduce TAVID, a unified framework that generates both interactive faces and conversational speech in a synchronized manner. TAVID integrates face and speech generation pipelines through two cross-modal mappers (i.e., a motion mapper and a speaker mapper), which enable bidirectional exchange of complementary information between the audio and visual modalities. We evaluate our system across four dimensions: talking face realism, listening head responsiveness, dyadic interaction fluency, and speech quality. Extensive experiments demonstrate the effectiveness of our approach across all these aspects. |
| title | TAVID: Text-Driven Audio-Visual Interactive Dialogue Generation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Audio and Speech Processing Image and Video Processing |
| url | https://arxiv.org/abs/2512.20296 |