Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.10424 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915492874158080 |
|---|---|
| author | Cao, Xu Shen, Yifan Lai, Bolin Ye, Wenqian Ma, Yunsheng Heintz, Joerg Chen, Jintai Huang, Meihuan Cao, Jianguo Zhang, Aidong Rehg, James M. |
| author_facet | Cao, Xu Shen, Yifan Lai, Bolin Ye, Wenqian Ma, Yunsheng Heintz, Joerg Chen, Jintai Huang, Meihuan Cao, Jianguo Zhang, Aidong Rehg, James M. |
| contents | Recently, Multimodal Large Language Models (MLLMs) and Vision Language Models (VLMs) have shown great promise in language-guided perceptual tasks such as recognition, segmentation, and object detection. However, their effectiveness in addressing visual cognition problems that require high-level multi-image reasoning and visual working memory is not well-established. One such challenge is matrix reasoning - the cognitive ability to discern relationships among patterns in a set of images and extrapolate to predict subsequent patterns. This skill is crucial during the early neurodevelopmental stages of children. Inspired by the matrix reasoning tasks in Raven's Progressive Matrices (RPM) and Wechsler Intelligence Scale for Children (WISC), we propose a new dataset MaRs-VQA to evaluate the visual cognition capability of MLLMs and compare their performance with existing human visual cognition studies. Based on the training data of MaRs-VQA, we also finetune a baseline model Qwen2-VCog with multi-stage cognition reasoning annotations. Our comparative experiments with different baselines reveal a gap between MLLMs and human intelligence, highlighting the visual cognitive limitations of current MLLMs. We believe that the public release of MaRs-VQA and the Qwen2-VCog baseline model will drive progress toward the next generation of MLLMs with human-like visual cognition abilities. MaRs-VQA is available at huggingface.co/datasets/IrohXu/VCog-Bench. The training code of Qwen2-VCog is available at github.com/IrohXu/Cognition-MLLM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_10424 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | What is the Visual Cognition Gap between Humans and Multimodal LLMs? Cao, Xu Shen, Yifan Lai, Bolin Ye, Wenqian Ma, Yunsheng Heintz, Joerg Chen, Jintai Huang, Meihuan Cao, Jianguo Zhang, Aidong Rehg, James M. Computer Vision and Pattern Recognition Artificial Intelligence 68T01 Recently, Multimodal Large Language Models (MLLMs) and Vision Language Models (VLMs) have shown great promise in language-guided perceptual tasks such as recognition, segmentation, and object detection. However, their effectiveness in addressing visual cognition problems that require high-level multi-image reasoning and visual working memory is not well-established. One such challenge is matrix reasoning - the cognitive ability to discern relationships among patterns in a set of images and extrapolate to predict subsequent patterns. This skill is crucial during the early neurodevelopmental stages of children. Inspired by the matrix reasoning tasks in Raven's Progressive Matrices (RPM) and Wechsler Intelligence Scale for Children (WISC), we propose a new dataset MaRs-VQA to evaluate the visual cognition capability of MLLMs and compare their performance with existing human visual cognition studies. Based on the training data of MaRs-VQA, we also finetune a baseline model Qwen2-VCog with multi-stage cognition reasoning annotations. Our comparative experiments with different baselines reveal a gap between MLLMs and human intelligence, highlighting the visual cognitive limitations of current MLLMs. We believe that the public release of MaRs-VQA and the Qwen2-VCog baseline model will drive progress toward the next generation of MLLMs with human-like visual cognition abilities. MaRs-VQA is available at huggingface.co/datasets/IrohXu/VCog-Bench. The training code of Qwen2-VCog is available at github.com/IrohXu/Cognition-MLLM. |
| title | What is the Visual Cognition Gap between Humans and Multimodal LLMs? |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence 68T01 |
| url | https://arxiv.org/abs/2406.10424 |