VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917984648298496 |
|---|---|
| author | Tanaka, Ryota Iki, Taichi Hasegawa, Taku Nishida, Kyosuke Saito, Kuniko Suzuki, Jun |
| author_facet | Tanaka, Ryota Iki, Taichi Hasegawa, Taku Nishida, Kyosuke Saito, Kuniko Suzuki, Jun |
| contents | We aim to develop a retrieval-augmented generation (RAG) framework that answers questions over a corpus of visually-rich documents presented in mixed modalities (e.g., charts, tables) and diverse formats (e.g., PDF, PPTX). In this paper, we introduce a new RAG framework, VDocRAG, which can directly understand varied documents and modalities in a unified image format to prevent missing information that occurs by parsing documents to obtain text. To improve the performance, we propose novel self-supervised pre-training tasks that adapt large vision-language models for retrieval by compressing visual information into dense token representations while aligning them with textual content in documents. Furthermore, we introduce OpenDocVQA, the first unified collection of open-domain document visual question answering datasets, encompassing diverse document types and formats. OpenDocVQA provides a comprehensive resource for training and evaluating retrieval and question answering models on visually-rich documents in an open-domain setting. Experiments show that VDocRAG substantially outperforms conventional text-based RAG and has strong generalization capability, highlighting the potential of an effective RAG paradigm for real-world documents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_09795 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents Tanaka, Ryota Iki, Taichi Hasegawa, Taku Nishida, Kyosuke Saito, Kuniko Suzuki, Jun Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Information Retrieval We aim to develop a retrieval-augmented generation (RAG) framework that answers questions over a corpus of visually-rich documents presented in mixed modalities (e.g., charts, tables) and diverse formats (e.g., PDF, PPTX). In this paper, we introduce a new RAG framework, VDocRAG, which can directly understand varied documents and modalities in a unified image format to prevent missing information that occurs by parsing documents to obtain text. To improve the performance, we propose novel self-supervised pre-training tasks that adapt large vision-language models for retrieval by compressing visual information into dense token representations while aligning them with textual content in documents. Furthermore, we introduce OpenDocVQA, the first unified collection of open-domain document visual question answering datasets, encompassing diverse document types and formats. OpenDocVQA provides a comprehensive resource for training and evaluating retrieval and question answering models on visually-rich documents in an open-domain setting. Experiments show that VDocRAG substantially outperforms conventional text-based RAG and has strong generalization capability, highlighting the potential of an effective RAG paradigm for real-world documents. |
| title | VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Information Retrieval |
| url | https://arxiv.org/abs/2504.09795 |