MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910673957552128 |
|---|---|
| author | Zhu, Fengbin Liu, Ziyang Ng, Xiang Yao Wu, Haohui Wang, Wenjie Feng, Fuli Wang, Chao Luan, Huanbo Chua, Tat Seng |
| author_facet | Zhu, Fengbin Liu, Ziyang Ng, Xiang Yao Wu, Haohui Wang, Wenjie Feng, Fuli Wang, Chao Luan, Huanbo Chua, Tat Seng |
| contents | Large Vision-Language Models (LVLMs) have achieved remarkable performance in many vision-language tasks, yet their capabilities in fine-grained visual understanding remain insufficiently evaluated. Existing benchmarks either contain limited fine-grained evaluation samples that are mixed with other data, or are confined to object-level assessments in natural images. To holistically assess LVLMs' fine-grained visual understanding capabilities, we propose using document images with multi-granularity and multi-modal information to supplement natural images. In this light, we construct MMDocBench, a benchmark with various OCR-free document understanding tasks for the evaluation of fine-grained visual perception and reasoning abilities. MMDocBench defines 15 main tasks with 4,338 QA pairs and 11,353 supporting regions, covering various document images such as research papers, receipts, financial reports, Wikipedia tables, charts, and infographics. Based on MMDocBench, we conduct extensive experiments using 13 open-source and 3 proprietary advanced LVLMs, assessing their strengths and weaknesses across different tasks and document image types. The benchmark, task instructions, and evaluation code will be made publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_21311 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding Zhu, Fengbin Liu, Ziyang Ng, Xiang Yao Wu, Haohui Wang, Wenjie Feng, Fuli Wang, Chao Luan, Huanbo Chua, Tat Seng Computer Vision and Pattern Recognition Artificial Intelligence Large Vision-Language Models (LVLMs) have achieved remarkable performance in many vision-language tasks, yet their capabilities in fine-grained visual understanding remain insufficiently evaluated. Existing benchmarks either contain limited fine-grained evaluation samples that are mixed with other data, or are confined to object-level assessments in natural images. To holistically assess LVLMs' fine-grained visual understanding capabilities, we propose using document images with multi-granularity and multi-modal information to supplement natural images. In this light, we construct MMDocBench, a benchmark with various OCR-free document understanding tasks for the evaluation of fine-grained visual perception and reasoning abilities. MMDocBench defines 15 main tasks with 4,338 QA pairs and 11,353 supporting regions, covering various document images such as research papers, receipts, financial reports, Wikipedia tables, charts, and infographics. Based on MMDocBench, we conduct extensive experiments using 13 open-source and 3 proprietary advanced LVLMs, assessing their strengths and weaknesses across different tasks and document image types. The benchmark, task instructions, and evaluation code will be made publicly available. |
| title | MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2410.21311 |