AlignGen: Boosting Personalized Image Generation with Cross-Modality Prior Alignment
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910971368308736 |
|---|---|
| author | Lin, Yiheng Zhao, Shifang Liu, Ting Qu, Xiaochao Liu, Luoqi Zhao, Yao Wei, Yunchao |
| author_facet | Lin, Yiheng Zhao, Shifang Liu, Ting Qu, Xiaochao Liu, Luoqi Zhao, Yao Wei, Yunchao |
| contents | Personalized image generation aims to integrate user-provided concepts into text-to-image models, enabling the generation of customized content based on a given prompt. Recent zero-shot approaches, particularly those leveraging diffusion transformers, incorporate reference image information through multi-modal attention mechanism. This integration allows the generated output to be influenced by both the textual prior from the prompt and the visual prior from the reference image. However, we observe that when the prompt and reference image are misaligned, the generated results exhibit a stronger bias toward the textual prior, leading to a significant loss of reference content. To address this issue, we propose AlignGen, a Cross-Modality Prior Alignment mechanism that enhances personalized image generation by: 1) introducing a learnable token to bridge the gap between the textual and visual priors, 2) incorporating a robust training strategy to ensure proper prior alignment, and 3) employing a selective cross-modal attention mask within the multi-modal attention mechanism to further align the priors. Experimental results demonstrate that AlignGen outperforms existing zero-shot methods and even surpasses popular test-time optimization approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_21911 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AlignGen: Boosting Personalized Image Generation with Cross-Modality Prior Alignment Lin, Yiheng Zhao, Shifang Liu, Ting Qu, Xiaochao Liu, Luoqi Zhao, Yao Wei, Yunchao Computer Vision and Pattern Recognition Personalized image generation aims to integrate user-provided concepts into text-to-image models, enabling the generation of customized content based on a given prompt. Recent zero-shot approaches, particularly those leveraging diffusion transformers, incorporate reference image information through multi-modal attention mechanism. This integration allows the generated output to be influenced by both the textual prior from the prompt and the visual prior from the reference image. However, we observe that when the prompt and reference image are misaligned, the generated results exhibit a stronger bias toward the textual prior, leading to a significant loss of reference content. To address this issue, we propose AlignGen, a Cross-Modality Prior Alignment mechanism that enhances personalized image generation by: 1) introducing a learnable token to bridge the gap between the textual and visual priors, 2) incorporating a robust training strategy to ensure proper prior alignment, and 3) employing a selective cross-modal attention mask within the multi-modal attention mechanism to further align the priors. Experimental results demonstrate that AlignGen outperforms existing zero-shot methods and even surpasses popular test-time optimization approaches. |
| title | AlignGen: Boosting Personalized Image Generation with Cross-Modality Prior Alignment |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.21911 |