Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.17571 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912044958089216 |
|---|---|
| author | Chen, Changyou Ding, Han Sisman, Bunyamin Xu, Yi Xie, Ouye Yao, Benjamin Z. Tran, Son Dinh Zeng, Belinda |
| author_facet | Chen, Changyou Ding, Han Sisman, Bunyamin Xu, Yi Xie, Ouye Yao, Benjamin Z. Tran, Son Dinh Zeng, Belinda |
| contents | Diffusion-based generative modeling has been achieving state-of-the-art results on various generation tasks. Most diffusion models, however, are limited to a single-generation modeling. Can we generalize diffusion models with the ability of multi-modal generative training for more generalizable modeling? In this paper, we propose a principled way to define a diffusion model by constructing a unified multi-modal diffusion model in a common diffusion space. We define the forward diffusion process to be driven by an information aggregation from multiple types of task-data, e.g., images for a generation task and labels for a classification task. In the reverse process, we enforce information sharing by parameterizing a shared backbone denoising network with additional modality-specific decoder heads. Such a structure can simultaneously learn to generate different types of multi-modal data with a multi-task loss, which is derived from a new multi-modal variational lower bound that generalizes the standard diffusion model. We propose several multimodal generation settings to verify our framework, including image transition, masked-image training, joint image-label and joint image-representation generative modeling. Extensive experimental results on ImageNet indicate the effectiveness of our framework for various multi-modal generative modeling, which we believe is an important research direction worthy of more future explorations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_17571 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Diffusion Models For Multi-Modal Generative Modeling Chen, Changyou Ding, Han Sisman, Bunyamin Xu, Yi Xie, Ouye Yao, Benjamin Z. Tran, Son Dinh Zeng, Belinda Computer Vision and Pattern Recognition Diffusion-based generative modeling has been achieving state-of-the-art results on various generation tasks. Most diffusion models, however, are limited to a single-generation modeling. Can we generalize diffusion models with the ability of multi-modal generative training for more generalizable modeling? In this paper, we propose a principled way to define a diffusion model by constructing a unified multi-modal diffusion model in a common diffusion space. We define the forward diffusion process to be driven by an information aggregation from multiple types of task-data, e.g., images for a generation task and labels for a classification task. In the reverse process, we enforce information sharing by parameterizing a shared backbone denoising network with additional modality-specific decoder heads. Such a structure can simultaneously learn to generate different types of multi-modal data with a multi-task loss, which is derived from a new multi-modal variational lower bound that generalizes the standard diffusion model. We propose several multimodal generation settings to verify our framework, including image transition, masked-image training, joint image-label and joint image-representation generative modeling. Extensive experimental results on ImageNet indicate the effectiveness of our framework for various multi-modal generative modeling, which we believe is an important research direction worthy of more future explorations. |
| title | Diffusion Models For Multi-Modal Generative Modeling |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.17571 |