On Improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915109244239872 |
|---|---|
| author | Ifriqi, Tariq Berrada Astolfi, Pietro Hall, Melissa Askari-Hemmat, Reyhane Benchetrit, Yohann Havasi, Marton Muckley, Matthew Alahari, Karteek Romero-Soriano, Adriana Verbeek, Jakob Drozdzal, Michal |
| author_facet | Ifriqi, Tariq Berrada Astolfi, Pietro Hall, Melissa Askari-Hemmat, Reyhane Benchetrit, Yohann Havasi, Marton Muckley, Matthew Alahari, Karteek Romero-Soriano, Adriana Verbeek, Jakob Drozdzal, Michal |
| contents | Large-scale training of latent diffusion models (LDMs) has enabled unprecedented quality in image generation. However, the key components of the best performing LDM training recipes are oftentimes not available to the research community, preventing apple-to-apple comparisons and hindering the validation of progress in the field. In this work, we perform an in-depth study of LDM training recipes focusing on the performance of models and their training efficiency. To ensure apple-to-apple comparisons, we re-implement five previously published models with their corresponding recipes. Through our study, we explore the effects of (i)~the mechanisms used to condition the generative model on semantic information (e.g., text prompt) and control metadata (e.g., crop size, random flip flag, etc.) on the model performance, and (ii)~the transfer of the representations learned on smaller and lower-resolution datasets to larger ones on the training efficiency and model performance. We then propose a novel conditioning mechanism that disentangles semantic and control metadata conditionings and sets a new state-of-the-art in class-conditional generation on the ImageNet-1k dataset -- with FID improvements of 7% on 256 and 8% on 512 resolutions -- as well as text-to-image generation on the CC12M dataset -- with FID improvements of 8% on 256 and 23% on 512 resolution. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_03177 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On Improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models Ifriqi, Tariq Berrada Astolfi, Pietro Hall, Melissa Askari-Hemmat, Reyhane Benchetrit, Yohann Havasi, Marton Muckley, Matthew Alahari, Karteek Romero-Soriano, Adriana Verbeek, Jakob Drozdzal, Michal Computer Vision and Pattern Recognition Artificial Intelligence Large-scale training of latent diffusion models (LDMs) has enabled unprecedented quality in image generation. However, the key components of the best performing LDM training recipes are oftentimes not available to the research community, preventing apple-to-apple comparisons and hindering the validation of progress in the field. In this work, we perform an in-depth study of LDM training recipes focusing on the performance of models and their training efficiency. To ensure apple-to-apple comparisons, we re-implement five previously published models with their corresponding recipes. Through our study, we explore the effects of (i)~the mechanisms used to condition the generative model on semantic information (e.g., text prompt) and control metadata (e.g., crop size, random flip flag, etc.) on the model performance, and (ii)~the transfer of the representations learned on smaller and lower-resolution datasets to larger ones on the training efficiency and model performance. We then propose a novel conditioning mechanism that disentangles semantic and control metadata conditionings and sets a new state-of-the-art in class-conditional generation on the ImageNet-1k dataset -- with FID improvements of 7% on 256 and 8% on 512 resolutions -- as well as text-to-image generation on the CC12M dataset -- with FID improvements of 8% on 256 and 23% on 512 resolution. |
| title | On Improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2411.03177 |