Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.05712 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913725149085696 |
|---|---|
| author | Höpner, Niklas Eshuijs, Leon Alivanistos, Dimitrios Zamprogno, Giacomo Tiddi, Ilaria |
| author_facet | Höpner, Niklas Eshuijs, Leon Alivanistos, Dimitrios Zamprogno, Giacomo Tiddi, Ilaria |
| contents | Foundation models are increasingly used in scientific research, but evaluating AI-generated scientific work remains challenging. While expert reviews are costly, large language models (LLMs) as proxy reviewers have proven to be unreliable. To address this, we investigate two automatic evaluation metrics, specifically citation count prediction and review score prediction. We parse all papers of OpenReview and augment each submission with its citation count, reference, and research hypothesis. Our findings reveal that citation count prediction is more viable than review score prediction, and predicting scores is more difficult purely from the research hypothesis than from the full paper. Furthermore, we show that a simple prediction model based solely on title and abstract outperforms LLM-based reviewers, though it still falls short of human-level consistency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_05712 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Automatic Evaluation Metrics for Artificially Generated Scientific Research Höpner, Niklas Eshuijs, Leon Alivanistos, Dimitrios Zamprogno, Giacomo Tiddi, Ilaria Computers and Society Artificial Intelligence Machine Learning Foundation models are increasingly used in scientific research, but evaluating AI-generated scientific work remains challenging. While expert reviews are costly, large language models (LLMs) as proxy reviewers have proven to be unreliable. To address this, we investigate two automatic evaluation metrics, specifically citation count prediction and review score prediction. We parse all papers of OpenReview and augment each submission with its citation count, reference, and research hypothesis. Our findings reveal that citation count prediction is more viable than review score prediction, and predicting scores is more difficult purely from the research hypothesis than from the full paper. Furthermore, we show that a simple prediction model based solely on title and abstract outperforms LLM-based reviewers, though it still falls short of human-level consistency. |
| title | Automatic Evaluation Metrics for Artificially Generated Scientific Research |
| topic | Computers and Society Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2503.05712 |