Not Search, But Scan: Benchmarking MLLMs on Scan-Oriented Academic Paper Reasoning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908921388597248 |
|---|---|
| author | Li, Rongjin Tang, Zichen Wang, Xianghe Hu, Xinyi Wang, Zhengyu Lu, Zhengyu Huang, Yiling Chen, Jiayuan Tan, Weisheng Liu, Jiacheng Yang, Zhongjun E, Haihong |
| author_facet | Li, Rongjin Tang, Zichen Wang, Xianghe Hu, Xinyi Wang, Zhengyu Lu, Zhengyu Huang, Yiling Chen, Jiayuan Tan, Weisheng Liu, Jiacheng Yang, Zhongjun E, Haihong |
| contents | With the rapid progress of multimodal large language models (MLLMs), AI already performs well at literature retrieval and certain reasoning tasks, serving as a capable assistant to human researchers, yet it remains far from autonomous research. The fundamental reason is that current work on academic paper reasoning is largely confined to a search-oriented paradigm centered on pre-specified targets, with reasoning grounded in relevance retrieval, which struggles to support researcher-style full-document understanding, reasoning, and verification. To bridge this gap, we propose \textbf{ScholScan}, a new benchmark for academic paper reasoning. ScholScan introduces a scan-oriented task setting that asks models to read and cross-check entire papers like human researchers, scanning the document to identify consistency issues. The benchmark comprises 1,800 carefully annotated questions drawn from nine error categories across 13 natural-science domains and 715 papers, and provides detailed annotations for evidence localization and reasoning traces, together with a unified evaluation protocol. We assessed 15 models across 24 input configurations and conducted a fine-grained analysis of MLLM capabilities for all error categories. Across the board, retrieval-augmented generation (RAG) methods yield no significant improvements, revealing systematic deficiencies of current MLLMs on scan-oriented tasks and underscoring the challenge posed by ScholScan. We expect ScholScan to be the leading and representative work of the scan-oriented task paradigm. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_28651 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Not Search, But Scan: Benchmarking MLLMs on Scan-Oriented Academic Paper Reasoning Li, Rongjin Tang, Zichen Wang, Xianghe Hu, Xinyi Wang, Zhengyu Lu, Zhengyu Huang, Yiling Chen, Jiayuan Tan, Weisheng Liu, Jiacheng Yang, Zhongjun E, Haihong Artificial Intelligence With the rapid progress of multimodal large language models (MLLMs), AI already performs well at literature retrieval and certain reasoning tasks, serving as a capable assistant to human researchers, yet it remains far from autonomous research. The fundamental reason is that current work on academic paper reasoning is largely confined to a search-oriented paradigm centered on pre-specified targets, with reasoning grounded in relevance retrieval, which struggles to support researcher-style full-document understanding, reasoning, and verification. To bridge this gap, we propose \textbf{ScholScan}, a new benchmark for academic paper reasoning. ScholScan introduces a scan-oriented task setting that asks models to read and cross-check entire papers like human researchers, scanning the document to identify consistency issues. The benchmark comprises 1,800 carefully annotated questions drawn from nine error categories across 13 natural-science domains and 715 papers, and provides detailed annotations for evidence localization and reasoning traces, together with a unified evaluation protocol. We assessed 15 models across 24 input configurations and conducted a fine-grained analysis of MLLM capabilities for all error categories. Across the board, retrieval-augmented generation (RAG) methods yield no significant improvements, revealing systematic deficiencies of current MLLMs on scan-oriented tasks and underscoring the challenge posed by ScholScan. We expect ScholScan to be the leading and representative work of the scan-oriented task paradigm. |
| title | Not Search, But Scan: Benchmarking MLLMs on Scan-Oriented Academic Paper Reasoning |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2603.28651 |