ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915611710324736 |
|---|---|
| author | Sharma, Manasi Zhang, Chen Bo Calvin Bandi, Chaithanya Wang, Clinton Aich, Ankit Nghiem, Huy Rabbani, Tahseen Htet, Ye Jang, Brian Basu, Sumana Balwani, Aishwarya Peskoff, Denis Ayestaran, Marcos Hendryx, Sean M. Kenstler, Brad Liu, Bing |
| author_facet | Sharma, Manasi Zhang, Chen Bo Calvin Bandi, Chaithanya Wang, Clinton Aich, Ankit Nghiem, Huy Rabbani, Tahseen Htet, Ye Jang, Brian Basu, Sumana Balwani, Aishwarya Peskoff, Denis Ayestaran, Marcos Hendryx, Sean M. Kenstler, Brad Liu, Bing |
| contents | Deep Research (DR) is an emerging agent application that leverages large language models (LLMs) to address open-ended queries. It requires the integration of several capabilities, including multi-step reasoning, cross-document synthesis, and the generation of evidence-backed, long-form answers. Evaluating DR remains challenging because responses are lengthy and diverse, admit many valid solutions, and often depend on dynamic information sources. We introduce ResearchRubrics, a standardized benchmark for DR built with over 2,800+ hours of human labor that pairs realistic, domain-diverse prompts with 2,500+ expert-written, fine-grained rubrics to assess factual grounding, reasoning soundness, and clarity. We also propose a new complexity framework for categorizing DR tasks along three axes: conceptual breadth, logical nesting, and exploration. In addition, we develop human and model-based evaluation protocols that measure rubric adherence for DR agents. We evaluate several state-of-the-art DR systems and find that even leading agents like Gemini's DR and OpenAI's DR achieve under 68% average compliance with our rubrics, primarily due to missed implicit context and inadequate reasoning about retrieved information. Our results highlight the need for robust, scalable assessment of deep research capabilities, to which end we release ResearchRubrics(including all prompts, rubrics, and evaluation code) to facilitate progress toward well-justified research assistants. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_07685 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents Sharma, Manasi Zhang, Chen Bo Calvin Bandi, Chaithanya Wang, Clinton Aich, Ankit Nghiem, Huy Rabbani, Tahseen Htet, Ye Jang, Brian Basu, Sumana Balwani, Aishwarya Peskoff, Denis Ayestaran, Marcos Hendryx, Sean M. Kenstler, Brad Liu, Bing Artificial Intelligence Computation and Language Machine Learning Deep Research (DR) is an emerging agent application that leverages large language models (LLMs) to address open-ended queries. It requires the integration of several capabilities, including multi-step reasoning, cross-document synthesis, and the generation of evidence-backed, long-form answers. Evaluating DR remains challenging because responses are lengthy and diverse, admit many valid solutions, and often depend on dynamic information sources. We introduce ResearchRubrics, a standardized benchmark for DR built with over 2,800+ hours of human labor that pairs realistic, domain-diverse prompts with 2,500+ expert-written, fine-grained rubrics to assess factual grounding, reasoning soundness, and clarity. We also propose a new complexity framework for categorizing DR tasks along three axes: conceptual breadth, logical nesting, and exploration. In addition, we develop human and model-based evaluation protocols that measure rubric adherence for DR agents. We evaluate several state-of-the-art DR systems and find that even leading agents like Gemini's DR and OpenAI's DR achieve under 68% average compliance with our rubrics, primarily due to missed implicit context and inadequate reasoning about retrieved information. Our results highlight the need for robust, scalable assessment of deep research capabilities, to which end we release ResearchRubrics(including all prompts, rubrics, and evaluation code) to facilitate progress toward well-justified research assistants. |
| title | ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents |
| topic | Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2511.07685 |