ANCHOR: LLM-driven Subject Conditioning for Text-to-Image Synthesis
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908991859195904 |
|---|---|
| author | Ramakrishnan, Aashish Anantha Huang, Sharon X. Lee, Dongwon |
| author_facet | Ramakrishnan, Aashish Anantha Huang, Sharon X. Lee, Dongwon |
| contents | Text-to-image (T2I) models have achieved remarkable progress in high-quality image synthesis, yet most benchmarks rely on simple, self-contained prompts, failing to capture the complexity of real-world captions. Human-written captions often involve multiple interacting subjects, rich contextual references, and abstractive phrasing, conditions under which current image-text encoders like CLIP struggle. To systematically study these deficiencies, we introduce ANCHOR, a large-scale dataset of 70K+ abstractive captions sourced from five major news media organizations. Analysis with ANCHOR reveals persistent failures in multi-subject understanding, context reasoning, and nuanced grounding. Motivated by these challenges, we propose Subject-Aware Fine-tuning (SAFE), which uses Large Language Models (LLMs) to extract key subjects and enhance their representation at the embedding-level. Experiments with contemporary models show that SAFE significantly improves image-caption consistency and human preference alignment, serving as a practical and scalable solution. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_10141 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ANCHOR: LLM-driven Subject Conditioning for Text-to-Image Synthesis Ramakrishnan, Aashish Anantha Huang, Sharon X. Lee, Dongwon Computer Vision and Pattern Recognition Computation and Language Multimedia 65D19 I.2.7; I.4.0 Text-to-image (T2I) models have achieved remarkable progress in high-quality image synthesis, yet most benchmarks rely on simple, self-contained prompts, failing to capture the complexity of real-world captions. Human-written captions often involve multiple interacting subjects, rich contextual references, and abstractive phrasing, conditions under which current image-text encoders like CLIP struggle. To systematically study these deficiencies, we introduce ANCHOR, a large-scale dataset of 70K+ abstractive captions sourced from five major news media organizations. Analysis with ANCHOR reveals persistent failures in multi-subject understanding, context reasoning, and nuanced grounding. Motivated by these challenges, we propose Subject-Aware Fine-tuning (SAFE), which uses Large Language Models (LLMs) to extract key subjects and enhance their representation at the embedding-level. Experiments with contemporary models show that SAFE significantly improves image-caption consistency and human preference alignment, serving as a practical and scalable solution. |
| title | ANCHOR: LLM-driven Subject Conditioning for Text-to-Image Synthesis |
| topic | Computer Vision and Pattern Recognition Computation and Language Multimedia 65D19 I.2.7; I.4.0 |
| url | https://arxiv.org/abs/2404.10141 |