ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909280146292736 |
|---|---|
| author | D'Arcy, Mike Ross, Alexis Bransom, Erin Kuehl, Bailey Bragg, Jonathan Hope, Tom Downey, Doug |
| author_facet | D'Arcy, Mike Ross, Alexis Bransom, Erin Kuehl, Bailey Bragg, Jonathan Hope, Tom Downey, Doug |
| contents | We introduce the task of automatically revising scientific papers based on peer feedback and release ARIES, a dataset of review comments and their corresponding paper edits. The data is drawn from real reviewer-author interactions from computer science, and we provide labels linking each reviewer comment to the specific paper edits made by the author in response. We automatically create a high-precision silver training set, as well as an expert-labeled test set that shows high inter-annotator agreement. In experiments with 10 models covering the state of the art, we find that they struggle even to identify which edits correspond to a comment -- especially when the relationship between the edit and the comment is indirect and requires reasoning to uncover. We also extensively analyze GPT-4's ability to generate edits given a comment and the original paper. We find that it often succeeds on a superficial level, but tends to rigidly follow the wording of the feedback rather than the underlying intent, and lacks technical details compared to human-written edits. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2306_12587 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews D'Arcy, Mike Ross, Alexis Bransom, Erin Kuehl, Bailey Bragg, Jonathan Hope, Tom Downey, Doug Computation and Language We introduce the task of automatically revising scientific papers based on peer feedback and release ARIES, a dataset of review comments and their corresponding paper edits. The data is drawn from real reviewer-author interactions from computer science, and we provide labels linking each reviewer comment to the specific paper edits made by the author in response. We automatically create a high-precision silver training set, as well as an expert-labeled test set that shows high inter-annotator agreement. In experiments with 10 models covering the state of the art, we find that they struggle even to identify which edits correspond to a comment -- especially when the relationship between the edit and the comment is indirect and requires reasoning to uncover. We also extensively analyze GPT-4's ability to generate edits given a comment and the original paper. We find that it often succeeds on a superficial level, but tends to rigidly follow the wording of the feedback rather than the underlying intent, and lacks technical details compared to human-written edits. |
| title | ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2306.12587 |