BloomVQA: Assessing Hierarchical Multi-modal Comprehension
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909220031430656 |
|---|---|
| author | Gong, Yunye Shrestha, Robik Claypoole, Jared Cogswell, Michael Ray, Arijit Kanan, Christopher Divakaran, Ajay |
| author_facet | Gong, Yunye Shrestha, Robik Claypoole, Jared Cogswell, Michael Ray, Arijit Kanan, Christopher Divakaran, Ajay |
| contents | We propose a novel VQA dataset, BloomVQA, to facilitate comprehensive evaluation of large vision-language models on comprehension tasks. Unlike current benchmarks that often focus on fact-based memorization and simple reasoning tasks without theoretical grounding, we collect multiple-choice samples based on picture stories that reflect different levels of comprehension, as laid out in Bloom's Taxonomy, a classic framework for learning assessment widely adopted in education research. Our data maps to a novel hierarchical graph representation which enables automatic data augmentation and novel measures characterizing model consistency. We perform graded evaluation and reliability analysis on recent multi-modal models. In comparison to low-level tasks, we observe decreased performance on tasks requiring advanced comprehension and cognitive skills with up to 38.0\% drop in VQA accuracy. In comparison to earlier models, GPT-4V demonstrates improved accuracy over all comprehension levels and shows a tendency of bypassing visual inputs especially for higher-level tasks. Current models also show consistency patterns misaligned with human comprehension in various scenarios, demonstrating the need for improvement based on theoretically-grounded criteria. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_12716 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | BloomVQA: Assessing Hierarchical Multi-modal Comprehension Gong, Yunye Shrestha, Robik Claypoole, Jared Cogswell, Michael Ray, Arijit Kanan, Christopher Divakaran, Ajay Computer Vision and Pattern Recognition Computation and Language Machine Learning We propose a novel VQA dataset, BloomVQA, to facilitate comprehensive evaluation of large vision-language models on comprehension tasks. Unlike current benchmarks that often focus on fact-based memorization and simple reasoning tasks without theoretical grounding, we collect multiple-choice samples based on picture stories that reflect different levels of comprehension, as laid out in Bloom's Taxonomy, a classic framework for learning assessment widely adopted in education research. Our data maps to a novel hierarchical graph representation which enables automatic data augmentation and novel measures characterizing model consistency. We perform graded evaluation and reliability analysis on recent multi-modal models. In comparison to low-level tasks, we observe decreased performance on tasks requiring advanced comprehension and cognitive skills with up to 38.0\% drop in VQA accuracy. In comparison to earlier models, GPT-4V demonstrates improved accuracy over all comprehension levels and shows a tendency of bypassing visual inputs especially for higher-level tasks. Current models also show consistency patterns misaligned with human comprehension in various scenarios, demonstrating the need for improvement based on theoretically-grounded criteria. |
| title | BloomVQA: Assessing Hierarchical Multi-modal Comprehension |
| topic | Computer Vision and Pattern Recognition Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2312.12716 |