Improving Compositional Attribute Binding in Text-to-Image Generative Models via Enhanced Text Embeddings
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913755431960576 |
|---|---|
| author | Zarei, Arman Rezaei, Keivan Basu, Samyadeep Saberi, Mehrdad Moayeri, Mazda Kattakinda, Priyatham Feizi, Soheil |
| author_facet | Zarei, Arman Rezaei, Keivan Basu, Samyadeep Saberi, Mehrdad Moayeri, Mazda Kattakinda, Priyatham Feizi, Soheil |
| contents | Text-to-image diffusion-based generative models have the stunning ability to generate photo-realistic images and achieve state-of-the-art low FID scores on challenging image generation benchmarks. However, one of the primary failure modes of these text-to-image generative models is in composing attributes, objects, and their associated relationships accurately into an image. In our paper, we investigate compositional attribute binding failures, where the model fails to correctly associate descriptive attributes (such as color, shape, or texture) with the corresponding objects in the generated images, and highlight that imperfect text conditioning with CLIP text-encoder is one of the primary reasons behind the inability of these models to generate high-fidelity compositional scenes. In particular, we show that (i) there exists an optimal text-embedding space that can generate highly coherent compositional scenes showing that the output space of the CLIP text-encoder is sub-optimal, and (ii) the final token embeddings in CLIP are erroneous as they often include attention contributions from unrelated tokens in compositional prompts. Our main finding shows that significant compositional improvements can be achieved (without harming the model's FID score) by fine-tuning only a simple and parameter-efficient linear projection on CLIP's representation space in Stable-Diffusion variants using a small set of compositional image-text pairs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_07844 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Improving Compositional Attribute Binding in Text-to-Image Generative Models via Enhanced Text Embeddings Zarei, Arman Rezaei, Keivan Basu, Samyadeep Saberi, Mehrdad Moayeri, Mazda Kattakinda, Priyatham Feizi, Soheil Computer Vision and Pattern Recognition Text-to-image diffusion-based generative models have the stunning ability to generate photo-realistic images and achieve state-of-the-art low FID scores on challenging image generation benchmarks. However, one of the primary failure modes of these text-to-image generative models is in composing attributes, objects, and their associated relationships accurately into an image. In our paper, we investigate compositional attribute binding failures, where the model fails to correctly associate descriptive attributes (such as color, shape, or texture) with the corresponding objects in the generated images, and highlight that imperfect text conditioning with CLIP text-encoder is one of the primary reasons behind the inability of these models to generate high-fidelity compositional scenes. In particular, we show that (i) there exists an optimal text-embedding space that can generate highly coherent compositional scenes showing that the output space of the CLIP text-encoder is sub-optimal, and (ii) the final token embeddings in CLIP are erroneous as they often include attention contributions from unrelated tokens in compositional prompts. Our main finding shows that significant compositional improvements can be achieved (without harming the model's FID score) by fine-tuning only a simple and parameter-efficient linear projection on CLIP's representation space in Stable-Diffusion variants using a small set of compositional image-text pairs. |
| title | Improving Compositional Attribute Binding in Text-to-Image Generative Models via Enhanced Text Embeddings |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.07844 |