THEMIS: Towards Holistic Evaluation of MLLMs for Scientific Paper Fraud Forensics
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908914751111168 |
|---|---|
| author | Ma, Tzu-Yen Zhang, Bo Tang, Zichen Ding, Junpeng Tian, Haolin Li, Yuanze Hao, Zhuodi Ding, Zixin Wang, Zirui Yu, Xinyu Peng, Shiyao Zhao, Yizhuo Jiang, Ruomeng Huang, Yiling Zhao, Peizhi Chen, Jiayuan Tan, Weisheng Gao, Haocheng Liu, Yang Liu, Jiacheng Yang, Zhongjun Huang, Jiayu E, Haihong |
| author_facet | Ma, Tzu-Yen Zhang, Bo Tang, Zichen Ding, Junpeng Tian, Haolin Li, Yuanze Hao, Zhuodi Ding, Zixin Wang, Zirui Yu, Xinyu Peng, Shiyao Zhao, Yizhuo Jiang, Ruomeng Huang, Yiling Zhao, Peizhi Chen, Jiayuan Tan, Weisheng Gao, Haocheng Liu, Yang Liu, Jiacheng Yang, Zhongjun Huang, Jiayu E, Haihong |
| contents | We present THEMIS, a novel multi-task benchmark designed to comprehensively evaluate multimodal large language models (MLLMs) on visual fraud reasoning within real-world academic scenarios. Compared to existing benchmarks, THEMIS introduces three major advances. (1) Real-World Scenarios and Complexity: Our benchmark comprises over 4,000 questions spanning seven scenarios, derived from authentic retracted-paper cases and carefully curated multimodal synthetic data. With 60.47% complex-texture images, THEMIS bridges the critical gap between existing benchmarks and the complexity of real-world academic fraud. (2) Fraud-Type Diversity and Granularity: THEMIS systematically covers five challenging fraud types and introduces 16 fine-grained manipulation operations. On average, each sample undergoes multiple stacked manipulation operations, with the diversity and difficulty of these manipulations demanding a high level of visual fraud reasoning from the models. (3) Multi-Dimensional Capability Evaluation: We establish a mapping from fraud types to five core visual fraud reasoning capabilities, thereby enabling an evaluation that reveals the distinct strengths and specific weaknesses of different models across these core capabilities. Experiments on 16 leading MLLMs show that even the best-performing model, GPT-5, achieves an overall performance of only 56.15%, demonstrating that our benchmark presents a stringent test. We expect THEMIS to advance the development of MLLMs for complex, real-world fraud reasoning tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_25089 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | THEMIS: Towards Holistic Evaluation of MLLMs for Scientific Paper Fraud Forensics Ma, Tzu-Yen Zhang, Bo Tang, Zichen Ding, Junpeng Tian, Haolin Li, Yuanze Hao, Zhuodi Ding, Zixin Wang, Zirui Yu, Xinyu Peng, Shiyao Zhao, Yizhuo Jiang, Ruomeng Huang, Yiling Zhao, Peizhi Chen, Jiayuan Tan, Weisheng Gao, Haocheng Liu, Yang Liu, Jiacheng Yang, Zhongjun Huang, Jiayu E, Haihong Computer Vision and Pattern Recognition We present THEMIS, a novel multi-task benchmark designed to comprehensively evaluate multimodal large language models (MLLMs) on visual fraud reasoning within real-world academic scenarios. Compared to existing benchmarks, THEMIS introduces three major advances. (1) Real-World Scenarios and Complexity: Our benchmark comprises over 4,000 questions spanning seven scenarios, derived from authentic retracted-paper cases and carefully curated multimodal synthetic data. With 60.47% complex-texture images, THEMIS bridges the critical gap between existing benchmarks and the complexity of real-world academic fraud. (2) Fraud-Type Diversity and Granularity: THEMIS systematically covers five challenging fraud types and introduces 16 fine-grained manipulation operations. On average, each sample undergoes multiple stacked manipulation operations, with the diversity and difficulty of these manipulations demanding a high level of visual fraud reasoning from the models. (3) Multi-Dimensional Capability Evaluation: We establish a mapping from fraud types to five core visual fraud reasoning capabilities, thereby enabling an evaluation that reveals the distinct strengths and specific weaknesses of different models across these core capabilities. Experiments on 16 leading MLLMs show that even the best-performing model, GPT-5, achieves an overall performance of only 56.15%, demonstrating that our benchmark presents a stringent test. We expect THEMIS to advance the development of MLLMs for complex, real-world fraud reasoning tasks. |
| title | THEMIS: Towards Holistic Evaluation of MLLMs for Scientific Paper Fraud Forensics |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2603.25089 |