Evaluating Retrieval-Augmented Generation Agents for Autonomous Scientific Discovery in Astrophysics
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911048332738560 |
|---|---|
| author | Xu, Xueqing Bolliet, Boris Dimitrov, Adrian Laverick, Andrew Villaescusa-Navarro, Francisco Xu, Licong Zubeldia, Íñigo |
| author_facet | Xu, Xueqing Bolliet, Boris Dimitrov, Adrian Laverick, Andrew Villaescusa-Navarro, Francisco Xu, Licong Zubeldia, Íñigo |
| contents | We evaluate 9 Retrieval Augmented Generation (RAG) agent configurations on 105 Cosmology Question-Answer (QA) pairs that we built specifically for this purpose.The RAG configurations are manually evaluated by a human expert, that is, a total of 945 generated answers were assessed. We find that currently the best RAG agent configuration is with OpenAI embedding and generative model, yielding 91.4\% accuracy. Using our human evaluation results we calibrate LLM-as-a-Judge (LLMaaJ) system which can be used as a robust proxy for human evaluation. These results allow us to systematically select the best RAG agent configuration for multi-agent system for autonomous scientific discovery in astrophysics (e.g., cmbagent presented in a companion paper) and provide us with an LLMaaJ system that can be scaled to thousands of cosmology QA pairs. We make our QA dataset, human evaluation results, RAG pipelines, and LLMaaJ system publicly available for further use by the astrophysics community. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_07155 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Evaluating Retrieval-Augmented Generation Agents for Autonomous Scientific Discovery in Astrophysics Xu, Xueqing Bolliet, Boris Dimitrov, Adrian Laverick, Andrew Villaescusa-Navarro, Francisco Xu, Licong Zubeldia, Íñigo Instrumentation and Methods for Astrophysics Cosmology and Nongalactic Astrophysics Artificial Intelligence We evaluate 9 Retrieval Augmented Generation (RAG) agent configurations on 105 Cosmology Question-Answer (QA) pairs that we built specifically for this purpose.The RAG configurations are manually evaluated by a human expert, that is, a total of 945 generated answers were assessed. We find that currently the best RAG agent configuration is with OpenAI embedding and generative model, yielding 91.4\% accuracy. Using our human evaluation results we calibrate LLM-as-a-Judge (LLMaaJ) system which can be used as a robust proxy for human evaluation. These results allow us to systematically select the best RAG agent configuration for multi-agent system for autonomous scientific discovery in astrophysics (e.g., cmbagent presented in a companion paper) and provide us with an LLMaaJ system that can be scaled to thousands of cosmology QA pairs. We make our QA dataset, human evaluation results, RAG pipelines, and LLMaaJ system publicly available for further use by the astrophysics community. |
| title | Evaluating Retrieval-Augmented Generation Agents for Autonomous Scientific Discovery in Astrophysics |
| topic | Instrumentation and Methods for Astrophysics Cosmology and Nongalactic Astrophysics Artificial Intelligence |
| url | https://arxiv.org/abs/2507.07155 |