MAPWise: Evaluating Vision-Language Models for Advanced Map Queries
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913487885697024 |
|---|---|
| author | Mukhopadhyay, Srija Rajgaria, Abhishek Khatiwada, Prerana Gupta, Vivek Roth, Dan |
| author_facet | Mukhopadhyay, Srija Rajgaria, Abhishek Khatiwada, Prerana Gupta, Vivek Roth, Dan |
| contents | Vision-language models (VLMs) excel at tasks requiring joint understanding of visual and linguistic information. A particularly promising yet under-explored application for these models lies in answering questions based on various kinds of maps. This study investigates the efficacy of VLMs in answering questions based on choropleth maps, which are widely used for data analysis and representation. To facilitate and encourage research in this area, we introduce a novel map-based question-answering benchmark, consisting of maps from three geographical regions (United States, India, China), each containing 1000 questions. Our benchmark incorporates 43 diverse question templates, requiring nuanced understanding of relative spatial relationships, intricate map features, and complex reasoning. It also includes maps with discrete and continuous values, encompassing variations in color-mapping, category ordering, and stylistic patterns, enabling comprehensive analysis. We evaluate the performance of multiple VLMs on this benchmark, highlighting gaps in their abilities and providing insights for improving such models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_00255 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MAPWise: Evaluating Vision-Language Models for Advanced Map Queries Mukhopadhyay, Srija Rajgaria, Abhishek Khatiwada, Prerana Gupta, Vivek Roth, Dan Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Graphics Human-Computer Interaction Vision-language models (VLMs) excel at tasks requiring joint understanding of visual and linguistic information. A particularly promising yet under-explored application for these models lies in answering questions based on various kinds of maps. This study investigates the efficacy of VLMs in answering questions based on choropleth maps, which are widely used for data analysis and representation. To facilitate and encourage research in this area, we introduce a novel map-based question-answering benchmark, consisting of maps from three geographical regions (United States, India, China), each containing 1000 questions. Our benchmark incorporates 43 diverse question templates, requiring nuanced understanding of relative spatial relationships, intricate map features, and complex reasoning. It also includes maps with discrete and continuous values, encompassing variations in color-mapping, category ordering, and stylistic patterns, enabling comprehensive analysis. We evaluate the performance of multiple VLMs on this benchmark, highlighting gaps in their abilities and providing insights for improving such models. |
| title | MAPWise: Evaluating Vision-Language Models for Advanced Map Queries |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Graphics Human-Computer Interaction |
| url | https://arxiv.org/abs/2409.00255 |