Do text-free diffusion models learn discriminative visual representations?
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910618508853248 |
|---|---|
| author | Mukhopadhyay, Soumik Gwilliam, Matthew Yamaguchi, Yosuke Agarwal, Vatsal Padmanabhan, Namitha Swaminathan, Archana Zhou, Tianyi Ohya, Jun Shrivastava, Abhinav |
| author_facet | Mukhopadhyay, Soumik Gwilliam, Matthew Yamaguchi, Yosuke Agarwal, Vatsal Padmanabhan, Namitha Swaminathan, Archana Zhou, Tianyi Ohya, Jun Shrivastava, Abhinav |
| contents | While many unsupervised learning models focus on one family of tasks, either generative or discriminative, we explore the possibility of a unified representation learner: a model which addresses both families of tasks simultaneously. We identify diffusion models, a state-of-the-art method for generative tasks, as a prime candidate. Such models involve training a U-Net to iteratively predict and remove noise, and the resulting model can synthesize high-fidelity, diverse, novel images. We find that the intermediate feature maps of the U-Net are diverse, discriminative feature representations. We propose a novel attention mechanism for pooling feature maps and further leverage this mechanism as DifFormer, a transformer feature fusion of features from different diffusion U-Net blocks and noise steps. We also develop DifFeed, a novel feedback mechanism tailored to diffusion. We find that diffusion models are better than GANs, and, with our fusion and feedback mechanisms, can compete with state-of-the-art unsupervised image representation learning methods for discriminative tasks - image classification with full and semi-supervision, transfer for fine-grained classification, object detection and segmentation, and semantic segmentation. Our project website (https://mgwillia.github.io/diffssl/) and code (https://github.com/soumik-kanad/diffssl) are available publicly. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_17921 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Do text-free diffusion models learn discriminative visual representations? Mukhopadhyay, Soumik Gwilliam, Matthew Yamaguchi, Yosuke Agarwal, Vatsal Padmanabhan, Namitha Swaminathan, Archana Zhou, Tianyi Ohya, Jun Shrivastava, Abhinav Computer Vision and Pattern Recognition While many unsupervised learning models focus on one family of tasks, either generative or discriminative, we explore the possibility of a unified representation learner: a model which addresses both families of tasks simultaneously. We identify diffusion models, a state-of-the-art method for generative tasks, as a prime candidate. Such models involve training a U-Net to iteratively predict and remove noise, and the resulting model can synthesize high-fidelity, diverse, novel images. We find that the intermediate feature maps of the U-Net are diverse, discriminative feature representations. We propose a novel attention mechanism for pooling feature maps and further leverage this mechanism as DifFormer, a transformer feature fusion of features from different diffusion U-Net blocks and noise steps. We also develop DifFeed, a novel feedback mechanism tailored to diffusion. We find that diffusion models are better than GANs, and, with our fusion and feedback mechanisms, can compete with state-of-the-art unsupervised image representation learning methods for discriminative tasks - image classification with full and semi-supervision, transfer for fine-grained classification, object detection and segmentation, and semantic segmentation. Our project website (https://mgwillia.github.io/diffssl/) and code (https://github.com/soumik-kanad/diffssl) are available publicly. |
| title | Do text-free diffusion models learn discriminative visual representations? |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2311.17921 |