Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.15031 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918028016353280 |
|---|---|
| author | Wu, Wenqing Xi, Haixu Zhang, Chengzhi |
| author_facet | Wu, Wenqing Xi, Haixu Zhang, Chengzhi |
| contents | Peer review is vital in academia for evaluating research quality. Top AI conferences use reviewer confidence scores to ensure review reliability, but existing studies lack fine-grained analysis of text-score consistency, potentially missing key details. This work assesses consistency at word, sentence, and aspect levels using deep learning and NLP conference review data. We employ deep learning to detect hedge sentences and aspects, then analyze report length, hedge word/sentence frequency, aspect mentions, and sentiment to evaluate text-score alignment. Correlation, significance, and regression tests examine confidence scores' impact on paper outcomes. Results show high text-score consistency across all levels, with regression revealing higher confidence scores correlate with paper rejection, validating expert assessments and peer review fairness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_15031 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Are the confidence scores of reviewers consistent with the review content? Evidence from top conference proceedings in AI Wu, Wenqing Xi, Haixu Zhang, Chengzhi Computation and Language Artificial Intelligence Human-Computer Interaction Information Retrieval Peer review is vital in academia for evaluating research quality. Top AI conferences use reviewer confidence scores to ensure review reliability, but existing studies lack fine-grained analysis of text-score consistency, potentially missing key details. This work assesses consistency at word, sentence, and aspect levels using deep learning and NLP conference review data. We employ deep learning to detect hedge sentences and aspects, then analyze report length, hedge word/sentence frequency, aspect mentions, and sentiment to evaluate text-score alignment. Correlation, significance, and regression tests examine confidence scores' impact on paper outcomes. Results show high text-score consistency across all levels, with regression revealing higher confidence scores correlate with paper rejection, validating expert assessments and peer review fairness. |
| title | Are the confidence scores of reviewers consistent with the review content? Evidence from top conference proceedings in AI |
| topic | Computation and Language Artificial Intelligence Human-Computer Interaction Information Retrieval |
| url | https://arxiv.org/abs/2505.15031 |