Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.16234 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915824020750336 |
|---|---|
| author | Moussa, Hanane Nour Da Silva, Patrick Queiroz Adu-Ampratwum, Daniel East, Alyson Lu, Zitong Puccetti, Nikki Xue, Mingyi Sun, Huan Majumder, Bodhisattwa Prasad Kumar, Sachin |
| author_facet | Moussa, Hanane Nour Da Silva, Patrick Queiroz Adu-Ampratwum, Daniel East, Alyson Lu, Zitong Puccetti, Nikki Xue, Mingyi Sun, Huan Majumder, Bodhisattwa Prasad Kumar, Sachin |
| contents | As AI tools become increasingly common for research ideation, robust evaluation is critical to ensure the validity and usefulness of generated ideas. We introduce ScholarEval, a retrieval augmented evaluation framework that assesses research ideas based on two fundamental criteria: soundness - the empirical validity of proposed methods based on existing literature, and contribution - the degree of advancement made by the idea across different dimensions relative to prior research. To evaluate ScholarEval, we introduce ScholarIdeas, the first expert-annotated dataset of multi-domain research ideas and reviews, comprised of 117 ideas across four disciplines: artificial intelligence, neuroscience, biochemistry, and ecology. Our evaluation shows that ScholarEval achieves significantly higher coverage of points mentioned in the human expert annotated rubrics in ScholarIdeas compared to all baselines. Furthermore, ScholarEval is consistently preferred over our strongest baseline o4-mini-deep-research, a reasoning and search-enabled agentic system by OpenAI, in terms of evaluation actionability, depth, and evidence support. Our large-scale user study also shows that ScholarEval significantly outperforms deep research in literature engagement, idea refinement, and usefulness. We openly release our code, dataset, and ScholarEval tool for the community to use and build on. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_16234 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ScholarEval: Research Idea Evaluation Grounded in Literature Moussa, Hanane Nour Da Silva, Patrick Queiroz Adu-Ampratwum, Daniel East, Alyson Lu, Zitong Puccetti, Nikki Xue, Mingyi Sun, Huan Majumder, Bodhisattwa Prasad Kumar, Sachin Artificial Intelligence Computation and Language Machine Learning As AI tools become increasingly common for research ideation, robust evaluation is critical to ensure the validity and usefulness of generated ideas. We introduce ScholarEval, a retrieval augmented evaluation framework that assesses research ideas based on two fundamental criteria: soundness - the empirical validity of proposed methods based on existing literature, and contribution - the degree of advancement made by the idea across different dimensions relative to prior research. To evaluate ScholarEval, we introduce ScholarIdeas, the first expert-annotated dataset of multi-domain research ideas and reviews, comprised of 117 ideas across four disciplines: artificial intelligence, neuroscience, biochemistry, and ecology. Our evaluation shows that ScholarEval achieves significantly higher coverage of points mentioned in the human expert annotated rubrics in ScholarIdeas compared to all baselines. Furthermore, ScholarEval is consistently preferred over our strongest baseline o4-mini-deep-research, a reasoning and search-enabled agentic system by OpenAI, in terms of evaluation actionability, depth, and evidence support. Our large-scale user study also shows that ScholarEval significantly outperforms deep research in literature engagement, idea refinement, and usefulness. We openly release our code, dataset, and ScholarEval tool for the community to use and build on. |
| title | ScholarEval: Research Idea Evaluation Grounded in Literature |
| topic | Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2510.16234 |