Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic View
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929360090431488 |
|---|---|
| author | Wang, Jin Dong, Shichao Zhu, Yapeng Yao, Kelu Zhao, Weidong Li, Chao Luo, Ping |
| author_facet | Wang, Jin Dong, Shichao Zhu, Yapeng Yao, Kelu Zhao, Weidong Li, Chao Luo, Ping |
| contents | Compositional reasoning capabilities are usually considered as fundamental skills to characterize human perception. Recent studies show that current Vision Language Models (VLMs) surprisingly lack sufficient knowledge with respect to such capabilities. To this end, we propose to thoroughly diagnose the composition representations encoded by VLMs, systematically revealing the potential cause for this weakness. Specifically, we propose evaluation methods from a novel game-theoretic view to assess the vulnerability of VLMs on different aspects of compositional understanding, e.g., relations and attributes. Extensive experimental results demonstrate and validate several insights to understand the incapabilities of VLMs on compositional reasoning, which provide useful and reliable guidance for future studies. The deliverables will be updated at https://vlms-compositionality-gametheory.github.io/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_17201 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic View Wang, Jin Dong, Shichao Zhu, Yapeng Yao, Kelu Zhao, Weidong Li, Chao Luo, Ping Computer Vision and Pattern Recognition Compositional reasoning capabilities are usually considered as fundamental skills to characterize human perception. Recent studies show that current Vision Language Models (VLMs) surprisingly lack sufficient knowledge with respect to such capabilities. To this end, we propose to thoroughly diagnose the composition representations encoded by VLMs, systematically revealing the potential cause for this weakness. Specifically, we propose evaluation methods from a novel game-theoretic view to assess the vulnerability of VLMs on different aspects of compositional understanding, e.g., relations and attributes. Extensive experimental results demonstrate and validate several insights to understand the incapabilities of VLMs on compositional reasoning, which provide useful and reliable guidance for future studies. The deliverables will be updated at https://vlms-compositionality-gametheory.github.io/. |
| title | Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic View |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.17201 |