Comparing Abstraction in Humans and Large Language Models Using Multimodal Serial Reproduction
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929235044597760 |
|---|---|
| author | Kumar, Sreejan Marjieh, Raja Zhang, Byron Campbell, Declan Hu, Michael Y. Bhatt, Umang Lake, Brenden Griffiths, Thomas L. |
| author_facet | Kumar, Sreejan Marjieh, Raja Zhang, Byron Campbell, Declan Hu, Michael Y. Bhatt, Umang Lake, Brenden Griffiths, Thomas L. |
| contents | Humans extract useful abstractions of the world from noisy sensory data. Serial reproduction allows us to study how people construe the world through a paradigm similar to the game of telephone, where one person observes a stimulus and reproduces it for the next to form a chain of reproductions. Past serial reproduction experiments typically employ a single sensory modality, but humans often communicate abstractions of the world to each other through language. To investigate the effect language on the formation of abstractions, we implement a novel multimodal serial reproduction framework by asking people who receive a visual stimulus to reproduce it in a linguistic format, and vice versa. We ran unimodal and multimodal chains with both humans and GPT-4 and find that adding language as a modality has a larger effect on human reproductions than GPT-4's. This suggests human visual and linguistic representations are more dissociable than those of GPT-4. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_03618 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Comparing Abstraction in Humans and Large Language Models Using Multimodal Serial Reproduction Kumar, Sreejan Marjieh, Raja Zhang, Byron Campbell, Declan Hu, Michael Y. Bhatt, Umang Lake, Brenden Griffiths, Thomas L. Artificial Intelligence Computation and Language Neurons and Cognition Humans extract useful abstractions of the world from noisy sensory data. Serial reproduction allows us to study how people construe the world through a paradigm similar to the game of telephone, where one person observes a stimulus and reproduces it for the next to form a chain of reproductions. Past serial reproduction experiments typically employ a single sensory modality, but humans often communicate abstractions of the world to each other through language. To investigate the effect language on the formation of abstractions, we implement a novel multimodal serial reproduction framework by asking people who receive a visual stimulus to reproduce it in a linguistic format, and vice versa. We ran unimodal and multimodal chains with both humans and GPT-4 and find that adding language as a modality has a larger effect on human reproductions than GPT-4's. This suggests human visual and linguistic representations are more dissociable than those of GPT-4. |
| title | Comparing Abstraction in Humans and Large Language Models Using Multimodal Serial Reproduction |
| topic | Artificial Intelligence Computation and Language Neurons and Cognition |
| url | https://arxiv.org/abs/2402.03618 |