Retrieval or Representation? Reassessing Benchmark Gaps in Multilingual and Visually Rich RAG
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912942705868800 |
|---|---|
| author | Asenov, Martin Benkirane, Kenza Goldwater, Dan Ghodsi, Aneiss |
| author_facet | Asenov, Martin Benkirane, Kenza Goldwater, Dan Ghodsi, Aneiss |
| contents | Retrieval-augmented generation (RAG) is a common way to ground language models in external documents and up-to-date information. Classical retrieval systems relied on lexical methods such as BM25, which rank documents by term overlap with corpus-level weighting. End-to-end multimodal retrievers trained on large query-document datasets claim substantial improvements over these approaches, especially for multilingual documents with complex visual layouts. We demonstrate that better document representation is the primary driver of benchmark improvements. By systematically varying transcription and preprocessing methods while holding the retrieval mechanism fixed, we demonstrate that BM25 can recover large gaps on multilingual and visual benchmarks. Our findings call for decomposed evaluation benchmarks that separately measure transcription and retrieval capabilities, enabling the field to correctly attribute progress and focus effort where it matters. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_04238 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Retrieval or Representation? Reassessing Benchmark Gaps in Multilingual and Visually Rich RAG Asenov, Martin Benkirane, Kenza Goldwater, Dan Ghodsi, Aneiss Computation and Language Retrieval-augmented generation (RAG) is a common way to ground language models in external documents and up-to-date information. Classical retrieval systems relied on lexical methods such as BM25, which rank documents by term overlap with corpus-level weighting. End-to-end multimodal retrievers trained on large query-document datasets claim substantial improvements over these approaches, especially for multilingual documents with complex visual layouts. We demonstrate that better document representation is the primary driver of benchmark improvements. By systematically varying transcription and preprocessing methods while holding the retrieval mechanism fixed, we demonstrate that BM25 can recover large gaps on multilingual and visual benchmarks. Our findings call for decomposed evaluation benchmarks that separately measure transcription and retrieval capabilities, enabling the field to correctly attribute progress and focus effort where it matters. |
| title | Retrieval or Representation? Reassessing Benchmark Gaps in Multilingual and Visually Rich RAG |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2603.04238 |