Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.06287 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916782642561024 |
|---|---|
| author | FutureSearch : Bosse, Nikos I. Evans, Jon Gambee, Robert G. Hnyk, Daniel Mühlbacher, Peter Phillips, Lawrence Schwarz, Dan Wildman, Jack |
| author_facet | FutureSearch : Bosse, Nikos I. Evans, Jon Gambee, Robert G. Hnyk, Daniel Mühlbacher, Peter Phillips, Lawrence Schwarz, Dan Wildman, Jack |
| contents | Amongst the most common use cases of modern AI is LLM chat with web search enabled. However, no direct evaluations of the quality of web research agents exist that control for the continually-changing web. We introduce Deep Research Bench, consisting of 89 multi-step web research task instances of varying difficulty across 8 diverse task categories, with the answers carefully worked out by skilled humans. We provide a "RetroSearch" environment with a large frozen set of scraped web pages, and demonstrate that offline "RetroSearch" agents perform comparably to "live web" agents, enabling reliable evaluations of models over time. We provide robust agent tooling and scaffolding to benchmark major LLMs as they are released, including "thinking" models like o3 and Gemini 2.5 Pro. We include automated evaluations of the lengthy agent traces to report progress over time in hallucinations, tool use, and forgetting. Finally, we evaluate the major web research products branded as "Deep Research", "Deep Search", "Search", or "Research." Results are available on a public leaderboard at https://drb.futuresearch.ai/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_06287 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Deep Research Bench: Evaluating AI Web Research Agents FutureSearch : Bosse, Nikos I. Evans, Jon Gambee, Robert G. Hnyk, Daniel Mühlbacher, Peter Phillips, Lawrence Schwarz, Dan Wildman, Jack Artificial Intelligence Amongst the most common use cases of modern AI is LLM chat with web search enabled. However, no direct evaluations of the quality of web research agents exist that control for the continually-changing web. We introduce Deep Research Bench, consisting of 89 multi-step web research task instances of varying difficulty across 8 diverse task categories, with the answers carefully worked out by skilled humans. We provide a "RetroSearch" environment with a large frozen set of scraped web pages, and demonstrate that offline "RetroSearch" agents perform comparably to "live web" agents, enabling reliable evaluations of models over time. We provide robust agent tooling and scaffolding to benchmark major LLMs as they are released, including "thinking" models like o3 and Gemini 2.5 Pro. We include automated evaluations of the lengthy agent traces to report progress over time in hallucinations, tool use, and forgetting. Finally, we evaluate the major web research products branded as "Deep Research", "Deep Search", "Search", or "Research." Results are available on a public leaderboard at https://drb.futuresearch.ai/. |
| title | Deep Research Bench: Evaluating AI Web Research Agents |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2506.06287 |