Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.12708 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914549632860160 |
|---|---|
| author | Jin, Yiqiao Zhao, Qinlin Wang, Yiyang Chen, Hao Zhu, Kaijie Xiao, Yijia Wang, Jindong |
| author_facet | Jin, Yiqiao Zhao, Qinlin Wang, Yiyang Chen, Hao Zhu, Kaijie Xiao, Yijia Wang, Jindong |
| contents | Peer review is fundamental to the integrity and advancement of scientific publication. Traditional methods of peer review analyses often rely on exploration and statistics of existing peer review data, which do not adequately address the multivariate nature of the process, account for the latent variables, and are further constrained by privacy concerns due to the sensitive nature of the data. We introduce AgentReview, the first large language model (LLM) based peer review simulation framework, which effectively disentangles the impacts of multiple latent factors and addresses the privacy issue. Our study reveals significant insights, including a notable 37.1% variation in paper decisions due to reviewers' biases, supported by sociological theories such as the social influence theory, altruism fatigue, and authority bias. We believe that this study could offer valuable insights to improve the design of peer review mechanisms. Our code is available at https://github.com/Ahren09/AgentReview. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_12708 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AgentReview: Exploring Peer Review Dynamics with LLM Agents Jin, Yiqiao Zhao, Qinlin Wang, Yiyang Chen, Hao Zhu, Kaijie Xiao, Yijia Wang, Jindong Computation and Language Peer review is fundamental to the integrity and advancement of scientific publication. Traditional methods of peer review analyses often rely on exploration and statistics of existing peer review data, which do not adequately address the multivariate nature of the process, account for the latent variables, and are further constrained by privacy concerns due to the sensitive nature of the data. We introduce AgentReview, the first large language model (LLM) based peer review simulation framework, which effectively disentangles the impacts of multiple latent factors and addresses the privacy issue. Our study reveals significant insights, including a notable 37.1% variation in paper decisions due to reviewers' biases, supported by sociological theories such as the social influence theory, altruism fatigue, and authority bias. We believe that this study could offer valuable insights to improve the design of peer review mechanisms. Our code is available at https://github.com/Ahren09/AgentReview. |
| title | AgentReview: Exploring Peer Review Dynamics with LLM Agents |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.12708 |