Unsupervised Keypoints from Pretrained Diffusion Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911883588534272 |
|---|---|
| author | Hedlin, Eric Sharma, Gopal Mahajan, Shweta He, Xingzhe Isack, Hossam Rhodin, Abhishek Kar Helge Tagliasacchi, Andrea Yi, Kwang Moo |
| author_facet | Hedlin, Eric Sharma, Gopal Mahajan, Shweta He, Xingzhe Isack, Hossam Rhodin, Abhishek Kar Helge Tagliasacchi, Andrea Yi, Kwang Moo |
| contents | Unsupervised learning of keypoints and landmarks has seen significant progress with the help of modern neural network architectures, but performance is yet to match the supervised counterpart, making their practicability questionable. We leverage the emergent knowledge within text-to-image diffusion models, towards more robust unsupervised keypoints. Our core idea is to find text embeddings that would cause the generative model to consistently attend to compact regions in images (i.e. keypoints). To do so, we simply optimize the text embedding such that the cross-attention maps within the denoising network are localized as Gaussians with small standard deviations. We validate our performance on multiple datasets: the CelebA, CUB-200-2011, Tai-Chi-HD, DeepFashion, and Human3.6m datasets. We achieve significantly improved accuracy, sometimes even outperforming supervised ones, particularly for data that is non-aligned and less curated. Our code is publicly available and can be found through our project page: https://ubc-vision.github.io/StableKeypoints/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_00065 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Unsupervised Keypoints from Pretrained Diffusion Models Hedlin, Eric Sharma, Gopal Mahajan, Shweta He, Xingzhe Isack, Hossam Rhodin, Abhishek Kar Helge Tagliasacchi, Andrea Yi, Kwang Moo Computer Vision and Pattern Recognition Unsupervised learning of keypoints and landmarks has seen significant progress with the help of modern neural network architectures, but performance is yet to match the supervised counterpart, making their practicability questionable. We leverage the emergent knowledge within text-to-image diffusion models, towards more robust unsupervised keypoints. Our core idea is to find text embeddings that would cause the generative model to consistently attend to compact regions in images (i.e. keypoints). To do so, we simply optimize the text embedding such that the cross-attention maps within the denoising network are localized as Gaussians with small standard deviations. We validate our performance on multiple datasets: the CelebA, CUB-200-2011, Tai-Chi-HD, DeepFashion, and Human3.6m datasets. We achieve significantly improved accuracy, sometimes even outperforming supervised ones, particularly for data that is non-aligned and less curated. Our code is publicly available and can be found through our project page: https://ubc-vision.github.io/StableKeypoints/ |
| title | Unsupervised Keypoints from Pretrained Diffusion Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.00065 |