Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.03930 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917129722265600 |
|---|---|
| author | Jia, Dongya Chen, Zhuo Chen, Jiawei Du, Chenpeng Wu, Jian Cong, Jian Zhuang, Xiaobin Li, Chumin Wei, Zhen Wang, Yuping Wang, Yuxuan |
| author_facet | Jia, Dongya Chen, Zhuo Chen, Jiawei Du, Chenpeng Wu, Jian Cong, Jian Zhuang, Xiaobin Li, Chumin Wei, Zhen Wang, Yuping Wang, Yuxuan |
| contents | Several recent studies have attempted to autoregressively generate continuous speech representations without discrete speech tokens by combining diffusion and autoregressive models, yet they often face challenges with excessive computational loads or suboptimal outcomes. In this work, we propose Diffusion Transformer Autoregressive Modeling (DiTAR), a patch-based autoregressive framework combining a language model with a diffusion transformer. This approach significantly enhances the efficacy of autoregressive models for continuous tokens and reduces computational demands. DiTAR utilizes a divide-and-conquer strategy for patch generation, where the language model processes aggregated patch embeddings and the diffusion transformer subsequently generates the next patch based on the output of the language model. For inference, we propose defining temperature as the time point of introducing noise during the reverse diffusion ODE to balance diversity and determinism. We also show in the extensive scaling analysis that DiTAR has superb scalability. In zero-shot speech generation, DiTAR achieves state-of-the-art performance in robustness, speaker similarity, and naturalness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_03930 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation Jia, Dongya Chen, Zhuo Chen, Jiawei Du, Chenpeng Wu, Jian Cong, Jian Zhuang, Xiaobin Li, Chumin Wei, Zhen Wang, Yuping Wang, Yuxuan Audio and Speech Processing Artificial Intelligence Computation and Language Machine Learning Sound Several recent studies have attempted to autoregressively generate continuous speech representations without discrete speech tokens by combining diffusion and autoregressive models, yet they often face challenges with excessive computational loads or suboptimal outcomes. In this work, we propose Diffusion Transformer Autoregressive Modeling (DiTAR), a patch-based autoregressive framework combining a language model with a diffusion transformer. This approach significantly enhances the efficacy of autoregressive models for continuous tokens and reduces computational demands. DiTAR utilizes a divide-and-conquer strategy for patch generation, where the language model processes aggregated patch embeddings and the diffusion transformer subsequently generates the next patch based on the output of the language model. For inference, we propose defining temperature as the time point of introducing noise during the reverse diffusion ODE to balance diversity and determinism. We also show in the extensive scaling analysis that DiTAR has superb scalability. In zero-shot speech generation, DiTAR achieves state-of-the-art performance in robustness, speaker similarity, and naturalness. |
| title | DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation |
| topic | Audio and Speech Processing Artificial Intelligence Computation and Language Machine Learning Sound |
| url | https://arxiv.org/abs/2502.03930 |