ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910222515175424 |
|---|---|
| author | Riehl, Kevin Marin, Andres L. Zacharof, Nikofors Wu, Fan Langer, Patrick Jakob, Robert Kouvelas, Anastasios Fontaras, Georgios Makridis, Michail A. |
| author_facet | Riehl, Kevin Marin, Andres L. Zacharof, Nikofors Wu, Fan Langer, Patrick Jakob, Robert Kouvelas, Anastasios Fontaras, Georgios Makridis, Michail A. |
| contents | Scientific peer review increasingly struggles to assess reproducibility at the scale and complexity of modern research output. Evaluating reproducibility requires reconstructing experimental dependencies, methodological choices, data flows, and result-generating procedures, which often exceeds what human reviewers can provide. Agentic Reproducibility Assessment (ARA) formalizes reproducibility assessment as a structured reasoning task over scientific documents. Given a paper, ARA extracts a directed workflow graph linking sources, methods, experiments, and outputs, then evaluates its reconstructability using structural and content-based scores for reproducibility assessments. Experiments on 213 ReScience C articles - the largest cross-domain benchmark of human-validated computational reproducibility studies considered to date - demonstrate ARA's generalizability and consistent workflow reconstruction and assessment across LLMs, model temperatures, and scientific domains. ARA achieves ~61% accuracy on three benchmarks, and the highest accuracy reported on ReproBench (60.71% vs. 36.84%) and GoldStandardDB (61.68% vs. 43.56%), highlighting its potential to complement human review at scale and enabling next-generation peer review. Code and Data available: https://github.com/AndresLaverdeMarin/agentic_reproducibility_assessment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_02651 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review Riehl, Kevin Marin, Andres L. Zacharof, Nikofors Wu, Fan Langer, Patrick Jakob, Robert Kouvelas, Anastasios Fontaras, Georgios Makridis, Michail A. Digital Libraries Machine Learning Scientific peer review increasingly struggles to assess reproducibility at the scale and complexity of modern research output. Evaluating reproducibility requires reconstructing experimental dependencies, methodological choices, data flows, and result-generating procedures, which often exceeds what human reviewers can provide. Agentic Reproducibility Assessment (ARA) formalizes reproducibility assessment as a structured reasoning task over scientific documents. Given a paper, ARA extracts a directed workflow graph linking sources, methods, experiments, and outputs, then evaluates its reconstructability using structural and content-based scores for reproducibility assessments. Experiments on 213 ReScience C articles - the largest cross-domain benchmark of human-validated computational reproducibility studies considered to date - demonstrate ARA's generalizability and consistent workflow reconstruction and assessment across LLMs, model temperatures, and scientific domains. ARA achieves ~61% accuracy on three benchmarks, and the highest accuracy reported on ReproBench (60.71% vs. 36.84%) and GoldStandardDB (61.68% vs. 43.56%), highlighting its potential to complement human review at scale and enabling next-generation peer review. Code and Data available: https://github.com/AndresLaverdeMarin/agentic_reproducibility_assessment. |
| title | ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review |
| topic | Digital Libraries Machine Learning |
| url | https://arxiv.org/abs/2605.02651 |