Enhancing Peer Review in Astronomy: A Machine Learning and Optimization Approach to Reviewer Assignments for ALMA
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913697817952256 |
|---|---|
| author | Carpenter, John M. Corvillón, Andrea Shah, Nihar B. |
| author_facet | Carpenter, John M. Corvillón, Andrea Shah, Nihar B. |
| contents | The increasing volume of papers and proposals that undergo peer review emphasizes the pressing need for greater automation to effectively manage the growing scale. In this study, we present the deployment and evaluation of machine learning and optimization techniques to assign proposals to reviewers that were developed for the Atacama Large Millimeter/submillimeter Array (ALMA) during the Cycle 10 Call for Proposals issued in 2023. Using topic modeling algorithms, we identify the proposal topics and assess reviewers' expertise based on their previous ALMA proposal submissions. We then apply an adapted version of the assignment optimization algorithm from PeerReview4All (Stelmakh et al. 2021) to maximize the alignment between proposal topics and reviewer expertise. Our evaluation shows a significant improvement in matching reviewer expertise: the median similarity score between the proposal topic and reviewer expertise increased by 51 percentage points compared to the previous cycle, and the percentage of reviewers reporting expertise in their assigned proposals rose by 20 percentage points. Furthermore, the assignment process proved highly effective in that no proposals required reassignment due to significant mismatches, resulting in a savings of 3 to 5 days of manual effort. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_10009 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhancing Peer Review in Astronomy: A Machine Learning and Optimization Approach to Reviewer Assignments for ALMA Carpenter, John M. Corvillón, Andrea Shah, Nihar B. Instrumentation and Methods for Astrophysics Artificial Intelligence Digital Libraries The increasing volume of papers and proposals that undergo peer review emphasizes the pressing need for greater automation to effectively manage the growing scale. In this study, we present the deployment and evaluation of machine learning and optimization techniques to assign proposals to reviewers that were developed for the Atacama Large Millimeter/submillimeter Array (ALMA) during the Cycle 10 Call for Proposals issued in 2023. Using topic modeling algorithms, we identify the proposal topics and assess reviewers' expertise based on their previous ALMA proposal submissions. We then apply an adapted version of the assignment optimization algorithm from PeerReview4All (Stelmakh et al. 2021) to maximize the alignment between proposal topics and reviewer expertise. Our evaluation shows a significant improvement in matching reviewer expertise: the median similarity score between the proposal topic and reviewer expertise increased by 51 percentage points compared to the previous cycle, and the percentage of reviewers reporting expertise in their assigned proposals rose by 20 percentage points. Furthermore, the assignment process proved highly effective in that no proposals required reassignment due to significant mismatches, resulting in a savings of 3 to 5 days of manual effort. |
| title | Enhancing Peer Review in Astronomy: A Machine Learning and Optimization Approach to Reviewer Assignments for ALMA |
| topic | Instrumentation and Methods for Astrophysics Artificial Intelligence Digital Libraries |
| url | https://arxiv.org/abs/2410.10009 |