Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2605.20668 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918513352900608 |
|---|---|
| author | Kim, Seungone Yoon, Dongkeun Gashteovski, Kiril Suk, Juyoung Baek, Jinheon Aggarwal, Pranjal Wu, Ian Zaverkin, Viktor Petkoski, Spase Schrider, Daniel R. Dukovski, Ilija Santini, Francesco Mitreska, Biljana Jeong, Yong Kwon, Kyeongha Sim, Young Min Manasova, Dragana Porto, Arthur Mojsoska, Biljana Takamoto, Makoto Shuntov, Marko Liu, Ruoqi Lee, Hyunjoo Jenny Dinç, Niyazi Ulas Jo, Yehhyun Han, Sunkyu Lee, Chungwoo Li, Huishan Tsai, Esther H. R. Simsek, Ergun Shafi, Khushboo Chung, Yeonseung Park, Jihye Shulevski, Aleksandar Christiansen, Henrik Son, Yoosang Knight, Elly Montoya, Amanda Ahn, Jeongyoun Langkammer, Christian Moon, Heera Yoon, Changwon Stikov, Nikola Jang, Mooseok Choi, Edward Kim, Junhan Jung, Yeon Sik Kim, Woo Youn Kim, Jae Kyoung Anjum, Ishraq Md Kim, Hyun Uk Bridges, Drew Lawrence, Carolin Yue, Xiang Oh, Alice Asai, Akari Welleck, Sean Neubig, Graham |
| author_facet | Kim, Seungone Yoon, Dongkeun Gashteovski, Kiril Suk, Juyoung Baek, Jinheon Aggarwal, Pranjal Wu, Ian Zaverkin, Viktor Petkoski, Spase Schrider, Daniel R. Dukovski, Ilija Santini, Francesco Mitreska, Biljana Jeong, Yong Kwon, Kyeongha Sim, Young Min Manasova, Dragana Porto, Arthur Mojsoska, Biljana Takamoto, Makoto Shuntov, Marko Liu, Ruoqi Lee, Hyunjoo Jenny Dinç, Niyazi Ulas Jo, Yehhyun Han, Sunkyu Lee, Chungwoo Li, Huishan Tsai, Esther H. R. Simsek, Ergun Shafi, Khushboo Chung, Yeonseung Park, Jihye Shulevski, Aleksandar Christiansen, Henrik Son, Yoosang Knight, Elly Montoya, Amanda Ahn, Jeongyoun Langkammer, Christian Moon, Heera Yoon, Changwon Stikov, Nikola Jang, Mooseok Choi, Edward Kim, Junhan Jung, Yeon Sik Kim, Woo Youn Kim, Jae Kyoung Anjum, Ishraq Md Kim, Hyun Uk Bridges, Drew Lawrence, Carolin Yue, Xiang Oh, Alice Asai, Akari Welleck, Sean Neubig, Graham |
| contents | With the advancement of AI capabilities, AI reviewers are beginning to be deployed in scientific peer review, yet their capability and credibility remain in question: many scientists simply view them as probabilistic systems without the expertise to evaluate research, while other researchers are more optimistic about their readiness without concrete evidence. Understanding what AI reviewers do well, where they fall short, and what challenges remain is essential. However, existing evaluations of AI reviewers have focused on whether their verdicts match human verdicts (e.g., score alignment, acceptance prediction), which is insufficient to characterize their capabilities and limits. In this paper, we close this gap through a large-scale expert annotation study, in which 45 domain scientists in Physical, Biological, and Health Sciences spent 469 hours rating 2,960 individual criticisms (each targeting one specific aspect of a paper) from human-written and AI-generated reviews of 82 Nature-family papers on correctness, significance, and sufficiency of evidence. On a composite of all three dimensions, a reviewing agent powered by GPT-5.2 scores above each paper's top-rated human reviewer (60.0% vs. 48.2%, p = 0.009), while all three AI reviewers (including Gemini 3.0 Pro and Claude Opus 4.5) exceed the lowest-rated human across every dimension. AI reviewers' accurate criticisms are also more often rated significant and well-evidenced, and surface a distinct 26% of issues no human raises. However, AI reviewers overlap far more than humans do (21% vs. 3% for cross-reviewer pairs), and exhibit 16 recurring weaknesses humans do not share, such as limited subfield knowledge, lack of long context management over multiple files, and overly critical stance on minor issues. Overall, our results position current AI reviewers as complements to, not substitutes for, human reviewers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_20668 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists Kim, Seungone Yoon, Dongkeun Gashteovski, Kiril Suk, Juyoung Baek, Jinheon Aggarwal, Pranjal Wu, Ian Zaverkin, Viktor Petkoski, Spase Schrider, Daniel R. Dukovski, Ilija Santini, Francesco Mitreska, Biljana Jeong, Yong Kwon, Kyeongha Sim, Young Min Manasova, Dragana Porto, Arthur Mojsoska, Biljana Takamoto, Makoto Shuntov, Marko Liu, Ruoqi Lee, Hyunjoo Jenny Dinç, Niyazi Ulas Jo, Yehhyun Han, Sunkyu Lee, Chungwoo Li, Huishan Tsai, Esther H. R. Simsek, Ergun Shafi, Khushboo Chung, Yeonseung Park, Jihye Shulevski, Aleksandar Christiansen, Henrik Son, Yoosang Knight, Elly Montoya, Amanda Ahn, Jeongyoun Langkammer, Christian Moon, Heera Yoon, Changwon Stikov, Nikola Jang, Mooseok Choi, Edward Kim, Junhan Jung, Yeon Sik Kim, Woo Youn Kim, Jae Kyoung Anjum, Ishraq Md Kim, Hyun Uk Bridges, Drew Lawrence, Carolin Yue, Xiang Oh, Alice Asai, Akari Welleck, Sean Neubig, Graham Computation and Language Artificial Intelligence Machine Learning With the advancement of AI capabilities, AI reviewers are beginning to be deployed in scientific peer review, yet their capability and credibility remain in question: many scientists simply view them as probabilistic systems without the expertise to evaluate research, while other researchers are more optimistic about their readiness without concrete evidence. Understanding what AI reviewers do well, where they fall short, and what challenges remain is essential. However, existing evaluations of AI reviewers have focused on whether their verdicts match human verdicts (e.g., score alignment, acceptance prediction), which is insufficient to characterize their capabilities and limits. In this paper, we close this gap through a large-scale expert annotation study, in which 45 domain scientists in Physical, Biological, and Health Sciences spent 469 hours rating 2,960 individual criticisms (each targeting one specific aspect of a paper) from human-written and AI-generated reviews of 82 Nature-family papers on correctness, significance, and sufficiency of evidence. On a composite of all three dimensions, a reviewing agent powered by GPT-5.2 scores above each paper's top-rated human reviewer (60.0% vs. 48.2%, p = 0.009), while all three AI reviewers (including Gemini 3.0 Pro and Claude Opus 4.5) exceed the lowest-rated human across every dimension. AI reviewers' accurate criticisms are also more often rated significant and well-evidenced, and surface a distinct 26% of issues no human raises. However, AI reviewers overlap far more than humans do (21% vs. 3% for cross-reviewer pairs), and exhibit 16 recurring weaknesses humans do not share, such as limited subfield knowledge, lack of long context management over multiple files, and overly critical stance on minor issues. Overall, our results position current AI reviewers as complements to, not substitutes for, human reviewers. |
| title | On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2605.20668 |