Pre-trained Language Models Do Not Help Auto-regressive Text-to-Image Generation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914956452036608 |
|---|---|
| author | Zhang, Yuhui McKinzie, Brandon Gan, Zhe Shankar, Vaishaal Toshev, Alexander |
| author_facet | Zhang, Yuhui McKinzie, Brandon Gan, Zhe Shankar, Vaishaal Toshev, Alexander |
| contents | Recent advances in image tokenizers, such as VQ-VAE, have enabled text-to-image generation using auto-regressive methods, similar to language modeling. However, these methods have yet to leverage pre-trained language models, despite their adaptability to various downstream tasks. In this work, we explore this gap by adapting a pre-trained language model for auto-regressive text-to-image generation, and find that pre-trained language models offer limited help. We provide a two-fold explanation by analyzing tokens from each modality. First, we demonstrate that image tokens possess significantly different semantics compared to text tokens, rendering pre-trained language models no more effective in modeling them than randomly initialized ones. Second, the text tokens in the image-text datasets are too simple compared to normal language model pre-training data, which causes the catastrophic degradation of language models' capability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_16201 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Pre-trained Language Models Do Not Help Auto-regressive Text-to-Image Generation Zhang, Yuhui McKinzie, Brandon Gan, Zhe Shankar, Vaishaal Toshev, Alexander Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning Recent advances in image tokenizers, such as VQ-VAE, have enabled text-to-image generation using auto-regressive methods, similar to language modeling. However, these methods have yet to leverage pre-trained language models, despite their adaptability to various downstream tasks. In this work, we explore this gap by adapting a pre-trained language model for auto-regressive text-to-image generation, and find that pre-trained language models offer limited help. We provide a two-fold explanation by analyzing tokens from each modality. First, we demonstrate that image tokens possess significantly different semantics compared to text tokens, rendering pre-trained language models no more effective in modeling them than randomly initialized ones. Second, the text tokens in the image-text datasets are too simple compared to normal language model pre-training data, which causes the catastrophic degradation of language models' capability. |
| title | Pre-trained Language Models Do Not Help Auto-regressive Text-to-Image Generation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2311.16201 |