Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.10855 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911012881432576 |
|---|---|
| author | Li, Yijiang Gao, Qingying Zhao, Tianwei Wang, Bingyang Sun, Haoran Lyu, Haiyun Hawkins, Robert D. Vasconcelos, Nuno Golan, Tal Luo, Dezhi Deng, Hokin |
| author_facet | Li, Yijiang Gao, Qingying Zhao, Tianwei Wang, Bingyang Sun, Haoran Lyu, Haiyun Hawkins, Robert D. Vasconcelos, Nuno Golan, Tal Luo, Dezhi Deng, Hokin |
| contents | While Multi-modal Large Language Models (MLLMs) demonstrate impressive abilities over high-level perception and reasoning, their robustness in the wild remains limited, often falling short on tasks that are intuitive and effortless for humans. We examine the hypothesis that these deficiencies stem from the absence of core knowledge--rudimentary cognitive abilities innate to humans from early childhood. To explore the core knowledge representation in MLLMs, we introduce CoreCognition, a large-scale benchmark encompassing 12 core knowledge concepts grounded in developmental cognitive science. We evaluate 230 models with 11 different prompts, leading to a total of 2,530 data points for analysis. Our experiments uncover four key findings, collectively demonstrating core knowledge deficits in MLLMs: they consistently underperform and show reduced, or even absent, scalability on low-level abilities relative to high-level ones. Finally, we propose Concept Hacking, a novel controlled evaluation method that reveals MLLMs fail to progress toward genuine core knowledge understanding, but instead rely on shortcut learning as they scale. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_10855 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Core Knowledge Deficits in Multi-Modal Language Models Li, Yijiang Gao, Qingying Zhao, Tianwei Wang, Bingyang Sun, Haoran Lyu, Haiyun Hawkins, Robert D. Vasconcelos, Nuno Golan, Tal Luo, Dezhi Deng, Hokin Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition While Multi-modal Large Language Models (MLLMs) demonstrate impressive abilities over high-level perception and reasoning, their robustness in the wild remains limited, often falling short on tasks that are intuitive and effortless for humans. We examine the hypothesis that these deficiencies stem from the absence of core knowledge--rudimentary cognitive abilities innate to humans from early childhood. To explore the core knowledge representation in MLLMs, we introduce CoreCognition, a large-scale benchmark encompassing 12 core knowledge concepts grounded in developmental cognitive science. We evaluate 230 models with 11 different prompts, leading to a total of 2,530 data points for analysis. Our experiments uncover four key findings, collectively demonstrating core knowledge deficits in MLLMs: they consistently underperform and show reduced, or even absent, scalability on low-level abilities relative to high-level ones. Finally, we propose Concept Hacking, a novel controlled evaluation method that reveals MLLMs fail to progress toward genuine core knowledge understanding, but instead rely on shortcut learning as they scale. |
| title | Core Knowledge Deficits in Multi-Modal Language Models |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.10855 |