PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908793059672064 |
|---|---|
| author | Tu, Songjun Ma, Yiwen Lin, Jiahao Zhang, Qichao Lan, Xiangyuan Li, Junfeng. Xu, Nan Li, Linjing Zhao, Dongbin |
| author_facet | Tu, Songjun Ma, Yiwen Lin, Jiahao Zhang, Qichao Lan, Xiangyuan Li, Junfeng. Xu, Nan Li, Linjing Zhao, Dongbin |
| contents | Large language models can generate fluent peer reviews, yet their assessments often lack sufficient critical rigor when substantive issues are subtle and distributed across a paper. In this paper, we introduce PaperAudit-Bench, which consists of two components: (1) PaperAudit-Dataset, an error dataset covering both errors identifiable within individual sections and those requiring cross-section reasoning, designed for controlled evaluation under long-context settings; and (2) PaperAudit-Review, an automated review framework that integrates structured error detection with evidence-aware review generation to support critical assessment. Experiments on PaperAudit-Bench reveal large variability in error detectability across models and detection depths, highlighting the difficulty of identifying such errors under long-context settings. Relative to representative automated reviewing baselines, incorporating explicit error detection into the review workflow produces systematically stricter and more discriminative evaluations, demonstrating its suitability for peer review. Finally, we show that the dataset supports training lightweight LLM detectors via SFT and RL, enabling effective error detection at reduced computational cost. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_19916 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review Tu, Songjun Ma, Yiwen Lin, Jiahao Zhang, Qichao Lan, Xiangyuan Li, Junfeng. Xu, Nan Li, Linjing Zhao, Dongbin Computation and Language 68T50 I.2.7; I.7 Large language models can generate fluent peer reviews, yet their assessments often lack sufficient critical rigor when substantive issues are subtle and distributed across a paper. In this paper, we introduce PaperAudit-Bench, which consists of two components: (1) PaperAudit-Dataset, an error dataset covering both errors identifiable within individual sections and those requiring cross-section reasoning, designed for controlled evaluation under long-context settings; and (2) PaperAudit-Review, an automated review framework that integrates structured error detection with evidence-aware review generation to support critical assessment. Experiments on PaperAudit-Bench reveal large variability in error detectability across models and detection depths, highlighting the difficulty of identifying such errors under long-context settings. Relative to representative automated reviewing baselines, incorporating explicit error detection into the review workflow produces systematically stricter and more discriminative evaluations, demonstrating its suitability for peer review. Finally, we show that the dataset supports training lightweight LLM detectors via SFT and RL, enabling effective error detection at reduced computational cost. |
| title | PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review |
| topic | Computation and Language 68T50 I.2.7; I.7 |
| url | https://arxiv.org/abs/2601.19916 |