Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2408.13860 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914923635802112 |
|---|---|
| author | Mathur, Suyash Vardhan Bafna, Jainit Sushil Kartik, Kunal Khandelwal, Harshita Shrivastava, Manish Gupta, Vivek Bansal, Mohit Roth, Dan |
| author_facet | Mathur, Suyash Vardhan Bafna, Jainit Sushil Kartik, Kunal Khandelwal, Harshita Shrivastava, Manish Gupta, Vivek Bansal, Mohit Roth, Dan |
| contents | Existing datasets for tabular question answering typically focus exclusively on text within cells. However, real-world data is inherently multimodal, often blending images such as symbols, faces, icons, patterns, and charts with textual content in tables. With the evolution of AI models capable of multimodal reasoning, it is pertinent to assess their efficacy in handling such structured data. This study investigates whether current AI models can perform knowledge-aware reasoning on multimodal structured data. We explore their ability to reason on tables that integrate both images and text, introducing MMTabQA, a new dataset designed for this purpose. Our experiments highlight substantial challenges for current AI models in effectively integrating and interpreting multiple text and image inputs, understanding visual context, and comparing visual content across images. These findings establish our dataset as a robust benchmark for advancing AI's comprehension and capabilities in analyzing multimodal structured data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_13860 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Knowledge-Aware Reasoning over Multimodal Semi-structured Tables Mathur, Suyash Vardhan Bafna, Jainit Sushil Kartik, Kunal Khandelwal, Harshita Shrivastava, Manish Gupta, Vivek Bansal, Mohit Roth, Dan Computation and Language Computer Vision and Pattern Recognition Existing datasets for tabular question answering typically focus exclusively on text within cells. However, real-world data is inherently multimodal, often blending images such as symbols, faces, icons, patterns, and charts with textual content in tables. With the evolution of AI models capable of multimodal reasoning, it is pertinent to assess their efficacy in handling such structured data. This study investigates whether current AI models can perform knowledge-aware reasoning on multimodal structured data. We explore their ability to reason on tables that integrate both images and text, introducing MMTabQA, a new dataset designed for this purpose. Our experiments highlight substantial challenges for current AI models in effectively integrating and interpreting multiple text and image inputs, understanding visual context, and comparing visual content across images. These findings establish our dataset as a robust benchmark for advancing AI's comprehension and capabilities in analyzing multimodal structured data. |
| title | Knowledge-Aware Reasoning over Multimodal Semi-structured Tables |
| topic | Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.13860 |