Object-Attribute Binding in Text-to-Image Generation: Evaluation and Control
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911848039710720 |
|---|---|
| author | Trusca, Maria Mihaela Nuyts, Wolf Thomm, Jonathan Honig, Robert Hofmann, Thomas Tuytelaars, Tinne Moens, Marie-Francine |
| author_facet | Trusca, Maria Mihaela Nuyts, Wolf Thomm, Jonathan Honig, Robert Hofmann, Thomas Tuytelaars, Tinne Moens, Marie-Francine |
| contents | Current diffusion models create photorealistic images given a text prompt as input but struggle to correctly bind attributes mentioned in the text to the right objects in the image. This is evidenced by our novel image-graph alignment model called EPViT (Edge Prediction Vision Transformer) for the evaluation of image-text alignment. To alleviate the above problem, we propose focused cross-attention (FCA) that controls the visual attention maps by syntactic constraints found in the input sentence. Additionally, the syntax structure of the prompt helps to disentangle the multimodal CLIP embeddings that are commonly used in T2I generation. The resulting DisCLIP embeddings and FCA are easily integrated in state-of-the-art diffusion models without additional training of these models. We show substantial improvements in T2I generation and especially its attribute-object binding on several datasets.\footnote{Code and data will be made available upon acceptance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_13766 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Object-Attribute Binding in Text-to-Image Generation: Evaluation and Control Trusca, Maria Mihaela Nuyts, Wolf Thomm, Jonathan Honig, Robert Hofmann, Thomas Tuytelaars, Tinne Moens, Marie-Francine Computer Vision and Pattern Recognition Current diffusion models create photorealistic images given a text prompt as input but struggle to correctly bind attributes mentioned in the text to the right objects in the image. This is evidenced by our novel image-graph alignment model called EPViT (Edge Prediction Vision Transformer) for the evaluation of image-text alignment. To alleviate the above problem, we propose focused cross-attention (FCA) that controls the visual attention maps by syntactic constraints found in the input sentence. Additionally, the syntax structure of the prompt helps to disentangle the multimodal CLIP embeddings that are commonly used in T2I generation. The resulting DisCLIP embeddings and FCA are easily integrated in state-of-the-art diffusion models without additional training of these models. We show substantial improvements in T2I generation and especially its attribute-object binding on several datasets.\footnote{Code and data will be made available upon acceptance. |
| title | Object-Attribute Binding in Text-to-Image Generation: Evaluation and Control |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2404.13766 |