Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.00230 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917740477939712 |
|---|---|
| author | Zhao, Juntu Deng, Junyu Ye, Yixin Li, Chongxuan Deng, Zhijie Wang, Dequan |
| author_facet | Zhao, Juntu Deng, Junyu Ye, Yixin Li, Chongxuan Deng, Zhijie Wang, Dequan |
| contents | Advancements in text-to-image diffusion models have broadened extensive downstream practical applications, but such models often encounter misalignment issues between text and image. Taking the generation of a combination of two disentangled concepts as an example, say given the prompt "a tea cup of iced coke", existing models usually generate a glass cup of iced coke because the iced coke usually co-occurs with the glass cup instead of the tea one during model training. The root of such misalignment is attributed to the confusion in the latent semantic space of text-to-image diffusion models, and hence we refer to the "a tea cup of iced coke" phenomenon as Latent Concept Misalignment (LC-Mis). We leverage large language models (LLMs) to thoroughly investigate the scope of LC-Mis, and develop an automated pipeline for aligning the latent semantics of diffusion models to text prompts. Empirical assessments confirm the effectiveness of our approach, substantially reducing LC-Mis errors and enhancing the robustness and versatility of text-to-image diffusion models. The code and dataset are here: https://github.com/RossoneriZhao/iced_coke. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_00230 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Lost in Translation: Latent Concept Misalignment in Text-to-Image Diffusion Models Zhao, Juntu Deng, Junyu Ye, Yixin Li, Chongxuan Deng, Zhijie Wang, Dequan Artificial Intelligence Computation and Language Advancements in text-to-image diffusion models have broadened extensive downstream practical applications, but such models often encounter misalignment issues between text and image. Taking the generation of a combination of two disentangled concepts as an example, say given the prompt "a tea cup of iced coke", existing models usually generate a glass cup of iced coke because the iced coke usually co-occurs with the glass cup instead of the tea one during model training. The root of such misalignment is attributed to the confusion in the latent semantic space of text-to-image diffusion models, and hence we refer to the "a tea cup of iced coke" phenomenon as Latent Concept Misalignment (LC-Mis). We leverage large language models (LLMs) to thoroughly investigate the scope of LC-Mis, and develop an automated pipeline for aligning the latent semantics of diffusion models to text prompts. Empirical assessments confirm the effectiveness of our approach, substantially reducing LC-Mis errors and enhancing the robustness and versatility of text-to-image diffusion models. The code and dataset are here: https://github.com/RossoneriZhao/iced_coke. |
| title | Lost in Translation: Latent Concept Misalignment in Text-to-Image Diffusion Models |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2408.00230 |