Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.13762 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915332199809024 |
|---|---|
| author | Kim, Gwanghyun Martinez, Alonso Su, Yu-Chuan Jou, Brendan Lezama, José Gupta, Agrim Yu, Lijun Jiang, Lu Jansen, Aren Walker, Jacob Somandepalli, Krishna |
| author_facet | Kim, Gwanghyun Martinez, Alonso Su, Yu-Chuan Jou, Brendan Lezama, José Gupta, Agrim Yu, Lijun Jiang, Lu Jansen, Aren Walker, Jacob Somandepalli, Krishna |
| contents | Training diffusion models for audiovisual sequences allows for a range of generation tasks by learning conditional distributions of various input-output combinations of the two modalities. Nevertheless, this strategy often requires training a separate model for each task which is expensive. Here, we propose a novel training approach to effectively learn arbitrary conditional distributions in the audiovisual space.Our key contribution lies in how we parameterize the diffusion timestep in the forward diffusion process. Instead of the standard fixed diffusion timestep, we propose applying variable diffusion timesteps across the temporal dimension and across modalities of the inputs. This formulation offers flexibility to introduce variable noise levels for various portions of the input, hence the term mixture of noise levels. We propose a transformer-based audiovisual latent diffusion model and show that it can be trained in a task-agnostic fashion using our approach to enable a variety of audiovisual generation tasks at inference time. Experiments demonstrate the versatility of our method in tackling cross-modal and multimodal interpolation tasks in the audiovisual space. Notably, our proposed approach surpasses baselines in generating temporally and perceptually consistent samples conditioned on the input. Project page: avdit2024.github.io |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_13762 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation Kim, Gwanghyun Martinez, Alonso Su, Yu-Chuan Jou, Brendan Lezama, José Gupta, Agrim Yu, Lijun Jiang, Lu Jansen, Aren Walker, Jacob Somandepalli, Krishna Computer Vision and Pattern Recognition Machine Learning Multimedia Sound Audio and Speech Processing Training diffusion models for audiovisual sequences allows for a range of generation tasks by learning conditional distributions of various input-output combinations of the two modalities. Nevertheless, this strategy often requires training a separate model for each task which is expensive. Here, we propose a novel training approach to effectively learn arbitrary conditional distributions in the audiovisual space.Our key contribution lies in how we parameterize the diffusion timestep in the forward diffusion process. Instead of the standard fixed diffusion timestep, we propose applying variable diffusion timesteps across the temporal dimension and across modalities of the inputs. This formulation offers flexibility to introduce variable noise levels for various portions of the input, hence the term mixture of noise levels. We propose a transformer-based audiovisual latent diffusion model and show that it can be trained in a task-agnostic fashion using our approach to enable a variety of audiovisual generation tasks at inference time. Experiments demonstrate the versatility of our method in tackling cross-modal and multimodal interpolation tasks in the audiovisual space. Notably, our proposed approach surpasses baselines in generating temporally and perceptually consistent samples conditioned on the input. Project page: avdit2024.github.io |
| title | A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation |
| topic | Computer Vision and Pattern Recognition Machine Learning Multimedia Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2405.13762 |