DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909961477423104 |
|---|---|
| author | Coelho, João Ning, Jingjie He, Jingyuan Mao, Kangrui Paladugu, Abhijay Setlur, Pranav Jin, Jiahe Callan, Jamie Magalhães, João Martins, Bruno Xiong, Chenyan |
| author_facet | Coelho, João Ning, Jingjie He, Jingyuan Mao, Kangrui Paladugu, Abhijay Setlur, Pranav Jin, Jiahe Callan, Jamie Magalhães, João Martins, Bruno Xiong, Chenyan |
| contents | Deep research systems represent an emerging class of agentic information retrieval methods that generate comprehensive and well-supported reports to complex queries. However, most existing frameworks rely on dynamic commercial search APIs, which pose reproducibility and transparency challenges in addition to their cost. To address these limitations, we introduce \textsc{DeepResearchGym} as an open-source sandbox that combines a reproducible search API with a rigorous evaluation protocol for benchmarking deep research systems. The API indexes large-scale public web corpora, namely ClueWeb22 and FineWeb, using a state-of-the-art dense retriever and approximate nearest neighbor search via DiskANN. It achieves lower latency than popular commercial APIs while ensuring stable document rankings across runs, and is free for research use. To evaluate deep research systems' outputs, we extend the Researchy Questions benchmark with automatic metrics through LLM-as-a-judge to measure alignment with users' information needs, retrieval faithfulness, and report quality. Experimental results show that systems integrated with~\textsc{DeepResearchGym} achieve performance comparable to those using commercial APIs, with performance rankings remaining consistent across evaluation metrics. A case study on short-answer search agents further demonstrates the sandbox's utility for cost-effective training, showing that models trained within the sandbox can generalize to commercial search. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_19253 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research Coelho, João Ning, Jingjie He, Jingyuan Mao, Kangrui Paladugu, Abhijay Setlur, Pranav Jin, Jiahe Callan, Jamie Magalhães, João Martins, Bruno Xiong, Chenyan Information Retrieval Deep research systems represent an emerging class of agentic information retrieval methods that generate comprehensive and well-supported reports to complex queries. However, most existing frameworks rely on dynamic commercial search APIs, which pose reproducibility and transparency challenges in addition to their cost. To address these limitations, we introduce \textsc{DeepResearchGym} as an open-source sandbox that combines a reproducible search API with a rigorous evaluation protocol for benchmarking deep research systems. The API indexes large-scale public web corpora, namely ClueWeb22 and FineWeb, using a state-of-the-art dense retriever and approximate nearest neighbor search via DiskANN. It achieves lower latency than popular commercial APIs while ensuring stable document rankings across runs, and is free for research use. To evaluate deep research systems' outputs, we extend the Researchy Questions benchmark with automatic metrics through LLM-as-a-judge to measure alignment with users' information needs, retrieval faithfulness, and report quality. Experimental results show that systems integrated with~\textsc{DeepResearchGym} achieve performance comparable to those using commercial APIs, with performance rankings remaining consistent across evaluation metrics. A case study on short-answer search agents further demonstrates the sandbox's utility for cost-effective training, showing that models trained within the sandbox can generalize to commercial search. |
| title | DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2505.19253 |