AV-Flow: Transforming Text to Audio-Visual Human-like Interactions
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910835042942976 |
|---|---|
| author | Chatziagapi, Aggelina Morency, Louis-Philippe Gong, Hongyu Zollhoefer, Michael Samaras, Dimitris Richard, Alexander |
| author_facet | Chatziagapi, Aggelina Morency, Louis-Philippe Gong, Hongyu Zollhoefer, Michael Samaras, Dimitris Richard, Alexander |
| contents | We introduce AV-Flow, an audio-visual generative model that animates photo-realistic 4D talking avatars given only text input. In contrast to prior work that assumes an existing speech signal, we synthesize speech and vision jointly. We demonstrate human-like speech synthesis, synchronized lip motion, lively facial expressions and head pose; all generated from just text characters. The core premise of our approach lies in the architecture of our two parallel diffusion transformers. Intermediate highway connections ensure communication between the audio and visual modalities, and thus, synchronized speech intonation and facial dynamics (e.g., eyebrow motion). Our model is trained with flow matching, leading to expressive results and fast inference. In case of dyadic conversations, AV-Flow produces an always-on avatar, that actively listens and reacts to the audio-visual input of a user. Through extensive experiments, we show that our method outperforms prior work, synthesizing natural-looking 4D talking avatars. Project page: https://aggelinacha.github.io/AV-Flow/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_13133 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AV-Flow: Transforming Text to Audio-Visual Human-like Interactions Chatziagapi, Aggelina Morency, Louis-Philippe Gong, Hongyu Zollhoefer, Michael Samaras, Dimitris Richard, Alexander Computer Vision and Pattern Recognition We introduce AV-Flow, an audio-visual generative model that animates photo-realistic 4D talking avatars given only text input. In contrast to prior work that assumes an existing speech signal, we synthesize speech and vision jointly. We demonstrate human-like speech synthesis, synchronized lip motion, lively facial expressions and head pose; all generated from just text characters. The core premise of our approach lies in the architecture of our two parallel diffusion transformers. Intermediate highway connections ensure communication between the audio and visual modalities, and thus, synchronized speech intonation and facial dynamics (e.g., eyebrow motion). Our model is trained with flow matching, leading to expressive results and fast inference. In case of dyadic conversations, AV-Flow produces an always-on avatar, that actively listens and reacts to the audio-visual input of a user. Through extensive experiments, we show that our method outperforms prior work, synthesizing natural-looking 4D talking avatars. Project page: https://aggelinacha.github.io/AV-Flow/ |
| title | AV-Flow: Transforming Text to Audio-Visual Human-like Interactions |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2502.13133 |