The Cognitive Capabilities of Generative AI: A Comparative Analysis with Human Benchmarks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917799866138624 |
|---|---|
| author | Galatzer-Levy, Isaac R. Munday, David McGiffin, Jed Liu, Xin Karmon, Danny Labzovsky, Ilia Moroshko, Rivka Zait, Amir McDuff, Daniel |
| author_facet | Galatzer-Levy, Isaac R. Munday, David McGiffin, Jed Liu, Xin Karmon, Danny Labzovsky, Ilia Moroshko, Rivka Zait, Amir McDuff, Daniel |
| contents | There is increasing interest in tracking the capabilities of general intelligence foundation models. This study benchmarks leading large language models and vision language models against human performance on the Wechsler Adult Intelligence Scale (WAIS-IV), a comprehensive, population-normed assessment of underlying human cognition and intellectual abilities, with a focus on the domains of VerbalComprehension (VCI), Working Memory (WMI), and Perceptual Reasoning (PRI). Most models demonstrated exceptional capabilities in the storage, retrieval, and manipulation of tokens such as arbitrary sequences of letters and numbers, with performance on the Working Memory Index (WMI) greater or equal to the 99.5th percentile when compared to human population normative ability. Performance on the Verbal Comprehension Index (VCI) which measures retrieval of acquired information, and linguistic understanding about the meaning of words and their relationships to each other, also demonstrated consistent performance at or above the 98th percentile. Despite these broad strengths, we observed consistently poor performance on the Perceptual Reasoning Index (PRI; range 0.1-10th percentile) from multimodal models indicating profound inability to interpret and reason on visual information. Smaller and older model versions consistently performed worse, indicating that training data, parameter count and advances in tuning are resulting in significant advances in cognitive ability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_07391 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Cognitive Capabilities of Generative AI: A Comparative Analysis with Human Benchmarks Galatzer-Levy, Isaac R. Munday, David McGiffin, Jed Liu, Xin Karmon, Danny Labzovsky, Ilia Moroshko, Rivka Zait, Amir McDuff, Daniel Artificial Intelligence There is increasing interest in tracking the capabilities of general intelligence foundation models. This study benchmarks leading large language models and vision language models against human performance on the Wechsler Adult Intelligence Scale (WAIS-IV), a comprehensive, population-normed assessment of underlying human cognition and intellectual abilities, with a focus on the domains of VerbalComprehension (VCI), Working Memory (WMI), and Perceptual Reasoning (PRI). Most models demonstrated exceptional capabilities in the storage, retrieval, and manipulation of tokens such as arbitrary sequences of letters and numbers, with performance on the Working Memory Index (WMI) greater or equal to the 99.5th percentile when compared to human population normative ability. Performance on the Verbal Comprehension Index (VCI) which measures retrieval of acquired information, and linguistic understanding about the meaning of words and their relationships to each other, also demonstrated consistent performance at or above the 98th percentile. Despite these broad strengths, we observed consistently poor performance on the Perceptual Reasoning Index (PRI; range 0.1-10th percentile) from multimodal models indicating profound inability to interpret and reason on visual information. Smaller and older model versions consistently performed worse, indicating that training data, parameter count and advances in tuning are resulting in significant advances in cognitive ability. |
| title | The Cognitive Capabilities of Generative AI: A Comparative Analysis with Human Benchmarks |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2410.07391 |