Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918114004828160 |
|---|---|
| author | Huybrechts, Goeric Ronanki, Srikanth Jayanthi, Sai Muralidhar Fitzgerald, Jack Veeravanallur, Srinivasan |
| author_facet | Huybrechts, Goeric Ronanki, Srikanth Jayanthi, Sai Muralidhar Fitzgerald, Jack Veeravanallur, Srinivasan |
| contents | The proliferation of multimodal Large Language Models has significantly advanced the ability to analyze and understand complex data inputs from different modalities. However, the processing of long documents remains under-explored, largely due to a lack of suitable benchmarks. To address this, we introduce Document Haystack, a comprehensive benchmark designed to evaluate the performance of Vision Language Models (VLMs) on long, visually complex documents. Document Haystack features documents ranging from 5 to 200 pages and strategically inserts pure text or multimodal text+image "needles" at various depths within the documents to challenge VLMs' retrieval capabilities. Comprising 400 document variants and a total of 8,250 questions, it is supported by an objective, automated evaluation framework. We detail the construction and characteristics of the Document Haystack dataset, present results from prominent VLMs and discuss potential research avenues in this area. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_15882 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark Huybrechts, Goeric Ronanki, Srikanth Jayanthi, Sai Muralidhar Fitzgerald, Jack Veeravanallur, Srinivasan Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning The proliferation of multimodal Large Language Models has significantly advanced the ability to analyze and understand complex data inputs from different modalities. However, the processing of long documents remains under-explored, largely due to a lack of suitable benchmarks. To address this, we introduce Document Haystack, a comprehensive benchmark designed to evaluate the performance of Vision Language Models (VLMs) on long, visually complex documents. Document Haystack features documents ranging from 5 to 200 pages and strategically inserts pure text or multimodal text+image "needles" at various depths within the documents to challenge VLMs' retrieval capabilities. Comprising 400 document variants and a total of 8,250 questions, it is supported by an objective, automated evaluation framework. We detail the construction and characteristics of the Document Haystack dataset, present results from prominent VLMs and discuss potential research avenues in this area. |
| title | Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2507.15882 |