Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.00324 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916891783593984 |
|---|---|
| author | Gao, Qingying Li, Yijiang Lyu, Haiyun Sun, Haoran Luo, Dezhi Deng, Hokin |
| author_facet | Gao, Qingying Li, Yijiang Lyu, Haiyun Sun, Haoran Luo, Dezhi Deng, Hokin |
| contents | Knowing others' intentions and taking others' perspectives are two core components of human intelligence that are considered to be instantiations of theory-of-mind. Infiltrating machines with these abilities is an important step towards building human-level artificial intelligence. Here, to investigate intentionality understanding and level-2 perspective-taking in Vision Language Models (VLMs), we constructed the IntentBench and PerspectBench, which together contains over 300 cognitive experiments grounded in real-world scenarios and classic cognitive tasks. We found VLMs achieving high performance on intentionality understanding but low performance on level-2 perspective-taking. This suggests a potential dissociation between simulation-based and theory-based theory-of-mind abilities in VLMs, highlighting the concern that they are not capable of using model-based reasoning to infer others' mental states. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_00324 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Vision Language Models See What You Want but not What You See Gao, Qingying Li, Yijiang Lyu, Haiyun Sun, Haoran Luo, Dezhi Deng, Hokin Artificial Intelligence Knowing others' intentions and taking others' perspectives are two core components of human intelligence that are considered to be instantiations of theory-of-mind. Infiltrating machines with these abilities is an important step towards building human-level artificial intelligence. Here, to investigate intentionality understanding and level-2 perspective-taking in Vision Language Models (VLMs), we constructed the IntentBench and PerspectBench, which together contains over 300 cognitive experiments grounded in real-world scenarios and classic cognitive tasks. We found VLMs achieving high performance on intentionality understanding but low performance on level-2 perspective-taking. This suggests a potential dissociation between simulation-based and theory-based theory-of-mind abilities in VLMs, highlighting the concern that they are not capable of using model-based reasoning to infer others' mental states. |
| title | Vision Language Models See What You Want but not What You See |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2410.00324 |