SIN-Bench: Tracing Native Evidence Chains in Long-Context Multimodal Scientific Interleaved Literature
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914256247586816 |
|---|---|
| author | Ren, Yiming Wang, Junjie Meng, Yuxin Shi, Yihang Lin, Zhiqiang Chu, Ruihang Xu, Yiran Li, Ziming Zhao, Yunfei Wang, Zihan Qiao, Yu Tang, Ruiming Liu, Minghao Yang, Yujiu |
| author_facet | Ren, Yiming Wang, Junjie Meng, Yuxin Shi, Yihang Lin, Zhiqiang Chu, Ruihang Xu, Yiran Li, Ziming Zhao, Yunfei Wang, Zihan Qiao, Yu Tang, Ruiming Liu, Minghao Yang, Yujiu |
| contents | Evaluating whether multimodal large language models truly understand long-form scientific papers remains challenging: answer-only metrics and synthetic "Needle-In-A-Haystack" tests often reward answer matching without requiring a causal, evidence-linked reasoning trace in the document. We propose the "Fish-in-the-Ocean" (FITO) paradigm, which requires models to construct explicit cross-modal evidence chains within native scientific documents. To operationalize FITO, we build SIN-Data, a scientific interleaved corpus that preserves the native interleaving of text and figures. On top of it, we construct SIN-Bench with four progressive tasks covering evidence discovery (SIN-Find), hypothesis verification (SIN-Verify), grounded QA (SIN-QA), and evidence-anchored synthesis (SIN-Summary). We further introduce "No Evidence, No Score", scoring predictions when grounded to verifiable anchors and diagnosing evidence quality via matching, relevance, and logic. Experiments on eight MLLMs show that grounding is the primary bottleneck: Gemini-3-pro achieves the best average overall score (0.573), while GPT-5 attains the highest SIN-QA answer accuracy (0.767) but underperforms on evidence-aligned overall scores, exposing a gap between correctness and traceable support. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_10108 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SIN-Bench: Tracing Native Evidence Chains in Long-Context Multimodal Scientific Interleaved Literature Ren, Yiming Wang, Junjie Meng, Yuxin Shi, Yihang Lin, Zhiqiang Chu, Ruihang Xu, Yiran Li, Ziming Zhao, Yunfei Wang, Zihan Qiao, Yu Tang, Ruiming Liu, Minghao Yang, Yujiu Computation and Language Artificial Intelligence Multimedia Evaluating whether multimodal large language models truly understand long-form scientific papers remains challenging: answer-only metrics and synthetic "Needle-In-A-Haystack" tests often reward answer matching without requiring a causal, evidence-linked reasoning trace in the document. We propose the "Fish-in-the-Ocean" (FITO) paradigm, which requires models to construct explicit cross-modal evidence chains within native scientific documents. To operationalize FITO, we build SIN-Data, a scientific interleaved corpus that preserves the native interleaving of text and figures. On top of it, we construct SIN-Bench with four progressive tasks covering evidence discovery (SIN-Find), hypothesis verification (SIN-Verify), grounded QA (SIN-QA), and evidence-anchored synthesis (SIN-Summary). We further introduce "No Evidence, No Score", scoring predictions when grounded to verifiable anchors and diagnosing evidence quality via matching, relevance, and logic. Experiments on eight MLLMs show that grounding is the primary bottleneck: Gemini-3-pro achieves the best average overall score (0.573), while GPT-5 attains the highest SIN-QA answer accuracy (0.767) but underperforms on evidence-aligned overall scores, exposing a gap between correctness and traceable support. |
| title | SIN-Bench: Tracing Native Evidence Chains in Long-Context Multimodal Scientific Interleaved Literature |
| topic | Computation and Language Artificial Intelligence Multimedia |
| url | https://arxiv.org/abs/2601.10108 |