DEFT: Efficient Fine-Tuning of Diffusion Models by Learning the Generalised $h$-transform
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913849130614784 |
|---|---|
| author | Denker, Alexander Vargas, Francisco Padhy, Shreyas Didi, Kieran Mathis, Simon Dutordoir, Vincent Barbano, Riccardo Mathieu, Emile Komorowska, Urszula Julia Lio, Pietro |
| author_facet | Denker, Alexander Vargas, Francisco Padhy, Shreyas Didi, Kieran Mathis, Simon Dutordoir, Vincent Barbano, Riccardo Mathieu, Emile Komorowska, Urszula Julia Lio, Pietro |
| contents | Generative modelling paradigms based on denoising diffusion processes have emerged as a leading candidate for conditional sampling in inverse problems. In many real-world applications, we often have access to large, expensively trained unconditional diffusion models, which we aim to exploit for improving conditional sampling. Most recent approaches are motivated heuristically and lack a unifying framework, obscuring connections between them. Further, they often suffer from issues such as being very sensitive to hyperparameters, being expensive to train or needing access to weights hidden behind a closed API. In this work, we unify conditional training and sampling using the mathematically well-understood Doob's h-transform. This new perspective allows us to unify many existing methods under a common umbrella. Under this framework, we propose DEFT (Doob's h-transform Efficient FineTuning), a new approach for conditional generation that simply fine-tunes a very small network to quickly learn the conditional $h$-transform, while keeping the larger unconditional network unchanged. DEFT is much faster than existing baselines while achieving state-of-the-art performance across a variety of linear and non-linear benchmarks. On image reconstruction tasks, we achieve speedups of up to 1.6$\times$, while having the best perceptual quality on natural images and reconstruction performance on medical images. Further, we also provide initial experiments on protein motif scaffolding and outperform reconstruction guidance methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_01781 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DEFT: Efficient Fine-Tuning of Diffusion Models by Learning the Generalised $h$-transform Denker, Alexander Vargas, Francisco Padhy, Shreyas Didi, Kieran Mathis, Simon Dutordoir, Vincent Barbano, Riccardo Mathieu, Emile Komorowska, Urszula Julia Lio, Pietro Machine Learning Generative modelling paradigms based on denoising diffusion processes have emerged as a leading candidate for conditional sampling in inverse problems. In many real-world applications, we often have access to large, expensively trained unconditional diffusion models, which we aim to exploit for improving conditional sampling. Most recent approaches are motivated heuristically and lack a unifying framework, obscuring connections between them. Further, they often suffer from issues such as being very sensitive to hyperparameters, being expensive to train or needing access to weights hidden behind a closed API. In this work, we unify conditional training and sampling using the mathematically well-understood Doob's h-transform. This new perspective allows us to unify many existing methods under a common umbrella. Under this framework, we propose DEFT (Doob's h-transform Efficient FineTuning), a new approach for conditional generation that simply fine-tunes a very small network to quickly learn the conditional $h$-transform, while keeping the larger unconditional network unchanged. DEFT is much faster than existing baselines while achieving state-of-the-art performance across a variety of linear and non-linear benchmarks. On image reconstruction tasks, we achieve speedups of up to 1.6$\times$, while having the best perceptual quality on natural images and reconstruction performance on medical images. Further, we also provide initial experiments on protein motif scaffolding and outperform reconstruction guidance methods. |
| title | DEFT: Efficient Fine-Tuning of Diffusion Models by Learning the Generalised $h$-transform |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2406.01781 |