UniForm: A Unified Multi-Task Diffusion Transformer for Audio-Video Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911040339443712 |
|---|---|
| author | Zhao, Lei Feng, Linfeng Ge, Dongxu Chen, Rujin Yi, Fangqiu Zhang, Chi Zhang, Xiao-Lei Li, Xuelong |
| author_facet | Zhao, Lei Feng, Linfeng Ge, Dongxu Chen, Rujin Yi, Fangqiu Zhang, Chi Zhang, Xiao-Lei Li, Xuelong |
| contents | With the rise of diffusion models, audio-video generation has been revolutionized. However, most existing methods rely on separate modules for each modality, with limited exploration of unified generative architectures. In addition, many are confined to a single task and small-scale datasets. To overcome these limitations, we introduce UniForm, a unified multi-task diffusion transformer that generates both audio and visual modalities in a shared latent space. By using a unified denoising network, UniForm captures the inherent correlations between sound and vision. Additionally, we propose task-specific noise schemes and task tokens, enabling the model to support multiple tasks with a single set of parameters, including video-to-audio, audio-to-video and text-to-audio-video generation. Furthermore, by leveraging large language models and a large-scale text-audio-video combined dataset, UniForm achieves greater generative diversity than prior approaches. Experiments show that UniForm achieves performance close to the state-of-the-art single-task models across three generation tasks, with generated content that is not only highly aligned with real-world data distributions but also enables more diverse and fine-grained generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_03897 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | UniForm: A Unified Multi-Task Diffusion Transformer for Audio-Video Generation Zhao, Lei Feng, Linfeng Ge, Dongxu Chen, Rujin Yi, Fangqiu Zhang, Chi Zhang, Xiao-Lei Li, Xuelong Multimedia Artificial Intelligence Computer Vision and Pattern Recognition Sound Audio and Speech Processing With the rise of diffusion models, audio-video generation has been revolutionized. However, most existing methods rely on separate modules for each modality, with limited exploration of unified generative architectures. In addition, many are confined to a single task and small-scale datasets. To overcome these limitations, we introduce UniForm, a unified multi-task diffusion transformer that generates both audio and visual modalities in a shared latent space. By using a unified denoising network, UniForm captures the inherent correlations between sound and vision. Additionally, we propose task-specific noise schemes and task tokens, enabling the model to support multiple tasks with a single set of parameters, including video-to-audio, audio-to-video and text-to-audio-video generation. Furthermore, by leveraging large language models and a large-scale text-audio-video combined dataset, UniForm achieves greater generative diversity than prior approaches. Experiments show that UniForm achieves performance close to the state-of-the-art single-task models across three generation tasks, with generated content that is not only highly aligned with real-world data distributions but also enables more diverse and fine-grained generation. |
| title | UniForm: A Unified Multi-Task Diffusion Transformer for Audio-Video Generation |
| topic | Multimedia Artificial Intelligence Computer Vision and Pattern Recognition Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2502.03897 |