DuoDiff: Accelerating Diffusion Models with a Dual-Backbone Approach
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909416548204544 |
|---|---|
| author | Fernández, Daniel Gallo Matişan, Răzvan-Andrei Muñoz, Alejandro Monroy Vasilcoiu, Ana-Maria Partyka, Janusz Veljković, Tin Hadži Jazbec, Metod |
| author_facet | Fernández, Daniel Gallo Matişan, Răzvan-Andrei Muñoz, Alejandro Monroy Vasilcoiu, Ana-Maria Partyka, Janusz Veljković, Tin Hadži Jazbec, Metod |
| contents | Diffusion models have achieved unprecedented performance in image generation, yet they suffer from slow inference due to their iterative sampling process. To address this, early-exiting has recently been proposed, where the depth of the denoising network is made adaptive based on the (estimated) difficulty of each sampling step. Here, we discover an interesting "phase transition" in the sampling process of current adaptive diffusion models: the denoising network consistently exits early during the initial sampling steps, until it suddenly switches to utilizing the full network. Based on this, we propose accelerating generation by employing a shallower denoising network in the initial sampling steps and a deeper network in the later steps. We demonstrate empirically that our dual-backbone approach, DuoDiff, outperforms existing early-exit diffusion methods in both inference speed and generation quality. Importantly, DuoDiff is easy to implement and complementary to existing approaches for accelerating diffusion. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_09633 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DuoDiff: Accelerating Diffusion Models with a Dual-Backbone Approach Fernández, Daniel Gallo Matişan, Răzvan-Andrei Muñoz, Alejandro Monroy Vasilcoiu, Ana-Maria Partyka, Janusz Veljković, Tin Hadži Jazbec, Metod Computer Vision and Pattern Recognition Diffusion models have achieved unprecedented performance in image generation, yet they suffer from slow inference due to their iterative sampling process. To address this, early-exiting has recently been proposed, where the depth of the denoising network is made adaptive based on the (estimated) difficulty of each sampling step. Here, we discover an interesting "phase transition" in the sampling process of current adaptive diffusion models: the denoising network consistently exits early during the initial sampling steps, until it suddenly switches to utilizing the full network. Based on this, we propose accelerating generation by employing a shallower denoising network in the initial sampling steps and a deeper network in the later steps. We demonstrate empirically that our dual-backbone approach, DuoDiff, outperforms existing early-exit diffusion methods in both inference speed and generation quality. Importantly, DuoDiff is easy to implement and complementary to existing approaches for accelerating diffusion. |
| title | DuoDiff: Accelerating Diffusion Models with a Dual-Backbone Approach |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.09633 |