ReviewScore: Misinformed Peer Review Detection with Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915870384586752 |
|---|---|
| author | Ryu, Hyun Jang, Doohyuk Lee, Hyemin S. Jeong, Joonhyun Kim, Gyeongman Cho, Donghyeon Chu, Gyouk Hwang, Minyeong Jang, Hyeongwon Kim, Changhun Kim, Haechan Kim, Jina Kim, Joowon Kim, Yoonjeon Lee, Kwanhyung Park, Chanjae Yun, Heecheol Betz, Gregor Yang, Eunho |
| author_facet | Ryu, Hyun Jang, Doohyuk Lee, Hyemin S. Jeong, Joonhyun Kim, Gyeongman Cho, Donghyeon Chu, Gyouk Hwang, Minyeong Jang, Hyeongwon Kim, Changhun Kim, Haechan Kim, Jina Kim, Joowon Kim, Yoonjeon Lee, Kwanhyung Park, Chanjae Yun, Heecheol Betz, Gregor Yang, Eunho |
| contents | Peer review serves as a backbone of academic research, but in most AI conferences, the review quality is degrading as the number of submissions explodes. To reliably detect low-quality reviews, we define misinformed review points as either "weaknesses" in a review that contain incorrect premises, or "questions" in a review that can be already answered by the paper. We verify that 15.2% of weaknesses and 26.4% of questions are misinformed and introduce ReviewScore indicating if a review point is misinformed. To evaluate the factuality of each premise of weaknesses, we propose an automated engine that reconstructs every explicit and implicit premise from a weakness. We build a human expert-annotated ReviewScore dataset to check the ability of LLMs to automate ReviewScore evaluation. Then, we measure human-model agreements on ReviewScore using eight current state-of-the-art LLMs. The models show F1 scores of 0.4--0.5 and kappa scores of 0.3--0.4, indicating moderate agreement but also suggesting that fully automating the evaluation remains challenging. A thorough disagreement analysis reveals that most errors are due to models' incorrect reasoning. We also prove that evaluating premise-level factuality shows significantly higher agreements than evaluating weakness-level factuality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_21679 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ReviewScore: Misinformed Peer Review Detection with Large Language Models Ryu, Hyun Jang, Doohyuk Lee, Hyemin S. Jeong, Joonhyun Kim, Gyeongman Cho, Donghyeon Chu, Gyouk Hwang, Minyeong Jang, Hyeongwon Kim, Changhun Kim, Haechan Kim, Jina Kim, Joowon Kim, Yoonjeon Lee, Kwanhyung Park, Chanjae Yun, Heecheol Betz, Gregor Yang, Eunho Computation and Language Peer review serves as a backbone of academic research, but in most AI conferences, the review quality is degrading as the number of submissions explodes. To reliably detect low-quality reviews, we define misinformed review points as either "weaknesses" in a review that contain incorrect premises, or "questions" in a review that can be already answered by the paper. We verify that 15.2% of weaknesses and 26.4% of questions are misinformed and introduce ReviewScore indicating if a review point is misinformed. To evaluate the factuality of each premise of weaknesses, we propose an automated engine that reconstructs every explicit and implicit premise from a weakness. We build a human expert-annotated ReviewScore dataset to check the ability of LLMs to automate ReviewScore evaluation. Then, we measure human-model agreements on ReviewScore using eight current state-of-the-art LLMs. The models show F1 scores of 0.4--0.5 and kappa scores of 0.3--0.4, indicating moderate agreement but also suggesting that fully automating the evaluation remains challenging. A thorough disagreement analysis reveals that most errors are due to models' incorrect reasoning. We also prove that evaluating premise-level factuality shows significantly higher agreements than evaluating weakness-level factuality. |
| title | ReviewScore: Misinformed Peer Review Detection with Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2509.21679 |