Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.19320 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909756845719552 |
|---|---|
| author | Chen, Ming Cui, Liyuan Zhang, Wenyuan Zhang, Haoxian Zhou, Yan Li, Xiaohan Tang, Songlin Liu, Jiwen Liao, Borui Chen, Hejia Liu, Xiaoqiang Wan, Pengfei |
| author_facet | Chen, Ming Cui, Liyuan Zhang, Wenyuan Zhang, Haoxian Zhou, Yan Li, Xiaohan Tang, Songlin Liu, Jiwen Liao, Borui Chen, Hejia Liu, Xiaoqiang Wan, Pengfei |
| contents | Recently, interactive digital human video generation has attracted widespread attention and achieved remarkable progress. However, building such a practical system that can interact with diverse input signals in real time remains challenging to existing methods, which often struggle with heavy computational cost and limited controllability. In this work, we introduce an autoregressive video generation framework that enables interactive multimodal control and low-latency extrapolation in a streaming manner. With minimal modifications to a standard large language model (LLM), our framework accepts multimodal condition encodings including audio, pose, and text, and outputs spatially and semantically coherent representations to guide the denoising process of a diffusion head. To support this, we construct a large-scale dialogue dataset of approximately 20,000 hours from multiple sources, providing rich conversational scenarios for training. We further introduce a deep compression autoencoder with up to 64$\times$ reduction ratio, which effectively alleviates the long-horizon inference burden of the autoregressive model. Extensive experiments on duplex conversation, multilingual human synthesis, and interactive world model highlight the advantages of our approach in low latency, high efficiency, and fine-grained multimodal controllability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_19320 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MIDAS: Multimodal Interactive Digital-humAn Synthesis via Real-time Autoregressive Video Generation Chen, Ming Cui, Liyuan Zhang, Wenyuan Zhang, Haoxian Zhou, Yan Li, Xiaohan Tang, Songlin Liu, Jiwen Liao, Borui Chen, Hejia Liu, Xiaoqiang Wan, Pengfei Computer Vision and Pattern Recognition Artificial Intelligence Recently, interactive digital human video generation has attracted widespread attention and achieved remarkable progress. However, building such a practical system that can interact with diverse input signals in real time remains challenging to existing methods, which often struggle with heavy computational cost and limited controllability. In this work, we introduce an autoregressive video generation framework that enables interactive multimodal control and low-latency extrapolation in a streaming manner. With minimal modifications to a standard large language model (LLM), our framework accepts multimodal condition encodings including audio, pose, and text, and outputs spatially and semantically coherent representations to guide the denoising process of a diffusion head. To support this, we construct a large-scale dialogue dataset of approximately 20,000 hours from multiple sources, providing rich conversational scenarios for training. We further introduce a deep compression autoencoder with up to 64$\times$ reduction ratio, which effectively alleviates the long-horizon inference burden of the autoregressive model. Extensive experiments on duplex conversation, multilingual human synthesis, and interactive world model highlight the advantages of our approach in low latency, high efficiency, and fine-grained multimodal controllability. |
| title | MIDAS: Multimodal Interactive Digital-humAn Synthesis via Real-time Autoregressive Video Generation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2508.19320 |