SG-Adapter: Enhancing Text-to-Image Generation with Scene Graph Guidance
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910458728939520 |
|---|---|
| author | Shen, Guibao Wang, Luozhou Lin, Jiantao Ge, Wenhang Zhang, Chaozhe Tao, Xin Zhang, Yuan Wan, Pengfei Wang, Zhongyuan Chen, Guangyong Li, Yijun Chen, Ying-Cong |
| author_facet | Shen, Guibao Wang, Luozhou Lin, Jiantao Ge, Wenhang Zhang, Chaozhe Tao, Xin Zhang, Yuan Wan, Pengfei Wang, Zhongyuan Chen, Guangyong Li, Yijun Chen, Ying-Cong |
| contents | Recent advancements in text-to-image generation have been propelled by the development of diffusion models and multi-modality learning. However, since text is typically represented sequentially in these models, it often falls short in providing accurate contextualization and structural control. So the generated images do not consistently align with human expectations, especially in complex scenarios involving multiple objects and relationships. In this paper, we introduce the Scene Graph Adapter(SG-Adapter), leveraging the structured representation of scene graphs to rectify inaccuracies in the original text embeddings. The SG-Adapter's explicit and non-fully connected graph representation greatly improves the fully connected, transformer-based text representations. This enhancement is particularly notable in maintaining precise correspondence in scenarios involving multiple relationships. To address the challenges posed by low-quality annotated datasets like Visual Genome, we have manually curated a highly clean, multi-relational scene graph-image paired dataset MultiRels. Furthermore, we design three metrics derived from GPT-4V to effectively and thoroughly measure the correspondence between images and scene graphs. Both qualitative and quantitative results validate the efficacy of our approach in controlling the correspondence in multiple relationships. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_15321 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SG-Adapter: Enhancing Text-to-Image Generation with Scene Graph Guidance Shen, Guibao Wang, Luozhou Lin, Jiantao Ge, Wenhang Zhang, Chaozhe Tao, Xin Zhang, Yuan Wan, Pengfei Wang, Zhongyuan Chen, Guangyong Li, Yijun Chen, Ying-Cong Computer Vision and Pattern Recognition Recent advancements in text-to-image generation have been propelled by the development of diffusion models and multi-modality learning. However, since text is typically represented sequentially in these models, it often falls short in providing accurate contextualization and structural control. So the generated images do not consistently align with human expectations, especially in complex scenarios involving multiple objects and relationships. In this paper, we introduce the Scene Graph Adapter(SG-Adapter), leveraging the structured representation of scene graphs to rectify inaccuracies in the original text embeddings. The SG-Adapter's explicit and non-fully connected graph representation greatly improves the fully connected, transformer-based text representations. This enhancement is particularly notable in maintaining precise correspondence in scenarios involving multiple relationships. To address the challenges posed by low-quality annotated datasets like Visual Genome, we have manually curated a highly clean, multi-relational scene graph-image paired dataset MultiRels. Furthermore, we design three metrics derived from GPT-4V to effectively and thoroughly measure the correspondence between images and scene graphs. Both qualitative and quantitative results validate the efficacy of our approach in controlling the correspondence in multiple relationships. |
| title | SG-Adapter: Enhancing Text-to-Image Generation with Scene Graph Guidance |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.15321 |