CleanDIFT: Diffusion Features without Noise
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909960973058048 |
|---|---|
| author | Stracke, Nick Baumann, Stefan Andreas Bauer, Kolja Fundel, Frank Ommer, Björn |
| author_facet | Stracke, Nick Baumann, Stefan Andreas Bauer, Kolja Fundel, Frank Ommer, Björn |
| contents | Internal features from large-scale pre-trained diffusion models have recently been established as powerful semantic descriptors for a wide range of downstream tasks. Works that use these features generally need to add noise to images before passing them through the model to obtain the semantic features, as the models do not offer the most useful features when given images with little to no noise. We show that this noise has a critical impact on the usefulness of these features that cannot be remedied by ensembling with different random noises. We address this issue by introducing a lightweight, unsupervised fine-tuning method that enables diffusion backbones to provide high-quality, noise-free semantic features. We show that these features readily outperform previous diffusion features by a wide margin in a wide variety of extraction setups and downstream tasks, offering better performance than even ensemble-based methods at a fraction of the cost. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_03439 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CleanDIFT: Diffusion Features without Noise Stracke, Nick Baumann, Stefan Andreas Bauer, Kolja Fundel, Frank Ommer, Björn Computer Vision and Pattern Recognition Internal features from large-scale pre-trained diffusion models have recently been established as powerful semantic descriptors for a wide range of downstream tasks. Works that use these features generally need to add noise to images before passing them through the model to obtain the semantic features, as the models do not offer the most useful features when given images with little to no noise. We show that this noise has a critical impact on the usefulness of these features that cannot be remedied by ensembling with different random noises. We address this issue by introducing a lightweight, unsupervised fine-tuning method that enables diffusion backbones to provide high-quality, noise-free semantic features. We show that these features readily outperform previous diffusion features by a wide margin in a wide variety of extraction setups and downstream tasks, offering better performance than even ensemble-based methods at a fraction of the cost. |
| title | CleanDIFT: Diffusion Features without Noise |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.03439 |