ARTIST: Improving the Generation of Text-rich Images with Disentangled Diffusion Models and Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913592888000512 |
|---|---|
| author | Zhang, Jianyi Zhou, Yufan Gu, Jiuxiang Wigington, Curtis Yu, Tong Chen, Yiran Sun, Tong Zhang, Ruiyi |
| author_facet | Zhang, Jianyi Zhou, Yufan Gu, Jiuxiang Wigington, Curtis Yu, Tong Chen, Yiran Sun, Tong Zhang, Ruiyi |
| contents | Diffusion models have demonstrated exceptional capabilities in generating a broad spectrum of visual content, yet their proficiency in rendering text is still limited: they often generate inaccurate characters or words that fail to blend well with the underlying image. To address these shortcomings, we introduce a novel framework named, ARTIST, which incorporates a dedicated textual diffusion model to focus on the learning of text structures specifically. Initially, we pretrain this textual model to capture the intricacies of text representation. Subsequently, we finetune a visual diffusion model, enabling it to assimilate textual structure information from the pretrained textual model. This disentangled architecture design and training strategy significantly enhance the text rendering ability of the diffusion models for text-rich image generation. Additionally, we leverage the capabilities of pretrained large language models to interpret user intentions better, contributing to improved generation quality. Empirical results on the MARIO-Eval benchmark underscore the effectiveness of the proposed method, showing an improvement of up to 15% in various metrics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_12044 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ARTIST: Improving the Generation of Text-rich Images with Disentangled Diffusion Models and Large Language Models Zhang, Jianyi Zhou, Yufan Gu, Jiuxiang Wigington, Curtis Yu, Tong Chen, Yiran Sun, Tong Zhang, Ruiyi Computer Vision and Pattern Recognition Diffusion models have demonstrated exceptional capabilities in generating a broad spectrum of visual content, yet their proficiency in rendering text is still limited: they often generate inaccurate characters or words that fail to blend well with the underlying image. To address these shortcomings, we introduce a novel framework named, ARTIST, which incorporates a dedicated textual diffusion model to focus on the learning of text structures specifically. Initially, we pretrain this textual model to capture the intricacies of text representation. Subsequently, we finetune a visual diffusion model, enabling it to assimilate textual structure information from the pretrained textual model. This disentangled architecture design and training strategy significantly enhance the text rendering ability of the diffusion models for text-rich image generation. Additionally, we leverage the capabilities of pretrained large language models to interpret user intentions better, contributing to improved generation quality. Empirical results on the MARIO-Eval benchmark underscore the effectiveness of the proposed method, showing an improvement of up to 15% in various metrics. |
| title | ARTIST: Improving the Generation of Text-rich Images with Disentangled Diffusion Models and Large Language Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.12044 |