Analyzing the Roles of Language and Vision in Learning from Limited Data
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913345651605504 |
|---|---|
| author | Chen, Allison Sucholutsky, Ilia Russakovsky, Olga Griffiths, Thomas L. |
| author_facet | Chen, Allison Sucholutsky, Ilia Russakovsky, Olga Griffiths, Thomas L. |
| contents | Does language help make sense of the visual world? How important is it to actually see the world rather than having it described with words? These basic questions about the nature of intelligence have been difficult to answer because we only had one example of an intelligent system -- humans -- and limited access to cases that isolated language or vision. However, the development of sophisticated Vision-Language Models (VLMs) by artificial intelligence researchers offers us new opportunities to explore the contributions that language and vision make to learning about the world. We ablate components from the cognitive architecture of these models to identify their contributions to learning new tasks from limited data. We find that a language model leveraging all components recovers a majority of a VLM's performance, despite its lack of visual input, and that language seems to allow this by providing access to prior knowledge and reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_19669 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Analyzing the Roles of Language and Vision in Learning from Limited Data Chen, Allison Sucholutsky, Ilia Russakovsky, Olga Griffiths, Thomas L. Machine Learning Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Does language help make sense of the visual world? How important is it to actually see the world rather than having it described with words? These basic questions about the nature of intelligence have been difficult to answer because we only had one example of an intelligent system -- humans -- and limited access to cases that isolated language or vision. However, the development of sophisticated Vision-Language Models (VLMs) by artificial intelligence researchers offers us new opportunities to explore the contributions that language and vision make to learning about the world. We ablate components from the cognitive architecture of these models to identify their contributions to learning new tasks from limited data. We find that a language model leveraging all components recovers a majority of a VLM's performance, despite its lack of visual input, and that language seems to allow this by providing access to prior knowledge and reasoning. |
| title | Analyzing the Roles of Language and Vision in Learning from Limited Data |
| topic | Machine Learning Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.19669 |