Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysis
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913525539012608 |
|---|---|
| author | Yu, Jianxiang Ding, Zichen Tan, Jiaqi Luo, Kangyang Weng, Zhenmin Gong, Chenghua Zeng, Long Cui, Renjing Han, Chengcheng Sun, Qiushi Wu, Zhiyong Lan, Yunshi Li, Xiang |
| author_facet | Yu, Jianxiang Ding, Zichen Tan, Jiaqi Luo, Kangyang Weng, Zhenmin Gong, Chenghua Zeng, Long Cui, Renjing Han, Chengcheng Sun, Qiushi Wu, Zhiyong Lan, Yunshi Li, Xiang |
| contents | In recent years, the rapid increase in scientific papers has overwhelmed traditional review mechanisms, resulting in varying quality of publications. Although existing methods have explored the capabilities of Large Language Models (LLMs) for automated scientific reviewing, their generated contents are often generic or partial. To address the issues above, we introduce an automated paper reviewing framework SEA. It comprises of three modules: Standardization, Evaluation, and Analysis, which are represented by models SEA-S, SEA-E, and SEA-A, respectively. Initially, SEA-S distills data standardization capabilities of GPT-4 for integrating multiple reviews for a paper. Then, SEA-E utilizes standardized data for fine-tuning, enabling it to generate constructive reviews. Finally, SEA-A introduces a new evaluation metric called mismatch score to assess the consistency between paper contents and reviews. Moreover, we design a self-correction strategy to enhance the consistency. Extensive experimental results on datasets collected from eight venues show that SEA can generate valuable insights for authors to improve their papers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_12857 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysis Yu, Jianxiang Ding, Zichen Tan, Jiaqi Luo, Kangyang Weng, Zhenmin Gong, Chenghua Zeng, Long Cui, Renjing Han, Chengcheng Sun, Qiushi Wu, Zhiyong Lan, Yunshi Li, Xiang Computation and Language Digital Libraries Information Retrieval In recent years, the rapid increase in scientific papers has overwhelmed traditional review mechanisms, resulting in varying quality of publications. Although existing methods have explored the capabilities of Large Language Models (LLMs) for automated scientific reviewing, their generated contents are often generic or partial. To address the issues above, we introduce an automated paper reviewing framework SEA. It comprises of three modules: Standardization, Evaluation, and Analysis, which are represented by models SEA-S, SEA-E, and SEA-A, respectively. Initially, SEA-S distills data standardization capabilities of GPT-4 for integrating multiple reviews for a paper. Then, SEA-E utilizes standardized data for fine-tuning, enabling it to generate constructive reviews. Finally, SEA-A introduces a new evaluation metric called mismatch score to assess the consistency between paper contents and reviews. Moreover, we design a self-correction strategy to enhance the consistency. Extensive experimental results on datasets collected from eight venues show that SEA can generate valuable insights for authors to improve their papers. |
| title | Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysis |
| topic | Computation and Language Digital Libraries Information Retrieval |
| url | https://arxiv.org/abs/2407.12857 |