A$^2$-LLM: An End-to-end Conversational Audio Avatar Large Language Model
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910012021932032 |
|---|---|
| author | Hu, Xiaolin Yuan, Hang Sang, Xinzhu Yan, Binbin Yu, Zhou Huang, Cong Chen, Kai |
| author_facet | Hu, Xiaolin Yuan, Hang Sang, Xinzhu Yan, Binbin Yu, Zhou Huang, Cong Chen, Kai |
| contents | Developing expressive and responsive conversational digital humans is a cornerstone of next-generation human-computer interaction. While large language models (LLMs) have significantly enhanced dialogue capabilities, most current systems still rely on cascaded architectures that connect independent modules. These pipelines are often plagued by accumulated errors, high latency, and poor real-time performance. Lacking access to the underlying conversational context, these pipelines inherently prioritize rigid lip-sync over emotional depth. To address these challenges, we propose A$^2$-LLM, an end-to-end conversational audio avatar large language model that jointly reasons about language, audio prosody, and 3D facial motion within a unified framework. To facilitate training, we introduce FLAME-QA, a high-quality multimodal dataset designed to align semantic intent with expressive facial dynamics within a QA format. By leveraging deep semantic understanding, A$^2$-LLM generates emotionally rich facial movements beyond simple lip-synchronization. Experimental results demonstrate that our system achieves superior emotional expressiveness while maintaining real-time efficiency (500 ms latency, 0.7 RTF). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_04913 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | A$^2$-LLM: An End-to-end Conversational Audio Avatar Large Language Model Hu, Xiaolin Yuan, Hang Sang, Xinzhu Yan, Binbin Yu, Zhou Huang, Cong Chen, Kai Machine Learning Artificial Intelligence Sound Developing expressive and responsive conversational digital humans is a cornerstone of next-generation human-computer interaction. While large language models (LLMs) have significantly enhanced dialogue capabilities, most current systems still rely on cascaded architectures that connect independent modules. These pipelines are often plagued by accumulated errors, high latency, and poor real-time performance. Lacking access to the underlying conversational context, these pipelines inherently prioritize rigid lip-sync over emotional depth. To address these challenges, we propose A$^2$-LLM, an end-to-end conversational audio avatar large language model that jointly reasons about language, audio prosody, and 3D facial motion within a unified framework. To facilitate training, we introduce FLAME-QA, a high-quality multimodal dataset designed to align semantic intent with expressive facial dynamics within a QA format. By leveraging deep semantic understanding, A$^2$-LLM generates emotionally rich facial movements beyond simple lip-synchronization. Experimental results demonstrate that our system achieves superior emotional expressiveness while maintaining real-time efficiency (500 ms latency, 0.7 RTF). |
| title | A$^2$-LLM: An End-to-end Conversational Audio Avatar Large Language Model |
| topic | Machine Learning Artificial Intelligence Sound |
| url | https://arxiv.org/abs/2602.04913 |