A Vision Check-up for Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914628821319680 |
|---|---|
| author | Sharma, Pratyusha Shaham, Tamar Rott Baradad, Manel Fu, Stephanie Rodriguez-Munoz, Adrian Duggal, Shivam Isola, Phillip Torralba, Antonio |
| author_facet | Sharma, Pratyusha Shaham, Tamar Rott Baradad, Manel Fu, Stephanie Rodriguez-Munoz, Adrian Duggal, Shivam Isola, Phillip Torralba, Antonio |
| contents | What does learning to model relationships between strings teach large language models (LLMs) about the visual world? We systematically evaluate LLMs' abilities to generate and recognize an assortment of visual concepts of increasing complexity and then demonstrate how a preliminary visual representation learning system can be trained using models of text. As language models lack the ability to consume or output visual information as pixels, we use code to represent images in our study. Although LLM-generated images do not look like natural images, results on image generation and the ability of models to correct these generated images indicate that precise modeling of strings can teach language models about numerous aspects of the visual world. Furthermore, experiments on self-supervised visual representation learning, utilizing images generated with text models, highlight the potential to train vision models capable of making semantic assessments of natural images using just LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_01862 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Vision Check-up for Language Models Sharma, Pratyusha Shaham, Tamar Rott Baradad, Manel Fu, Stephanie Rodriguez-Munoz, Adrian Duggal, Shivam Isola, Phillip Torralba, Antonio Computer Vision and Pattern Recognition Computation and Language Machine Learning What does learning to model relationships between strings teach large language models (LLMs) about the visual world? We systematically evaluate LLMs' abilities to generate and recognize an assortment of visual concepts of increasing complexity and then demonstrate how a preliminary visual representation learning system can be trained using models of text. As language models lack the ability to consume or output visual information as pixels, we use code to represent images in our study. Although LLM-generated images do not look like natural images, results on image generation and the ability of models to correct these generated images indicate that precise modeling of strings can teach language models about numerous aspects of the visual world. Furthermore, experiments on self-supervised visual representation learning, utilizing images generated with text models, highlight the potential to train vision models capable of making semantic assessments of natural images using just LLMs. |
| title | A Vision Check-up for Language Models |
| topic | Computer Vision and Pattern Recognition Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2401.01862 |