DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912957142663168 |
|---|---|
| author | Han, Janghoon Kim, Heegyu Lee, Changho Lee, Dahm Park, Min Hyung Song, Hosung Choi, Stanley Jungkyu Lee, Moontae Lee, Honglak |
| author_facet | Han, Janghoon Kim, Heegyu Lee, Changho Lee, Dahm Park, Min Hyung Song, Hosung Choi, Stanley Jungkyu Lee, Moontae Lee, Honglak |
| contents | Recent advances in large language models have enabled deep research systems that generate expert-level reports through multi-step reasoning and evidence-based synthesis. However, evaluating such reports remains challenging: report quality is multifaceted, making it difficult to determine what to assess and by what criteria; LLM-based judges may miss errors that require domain expertise to identify; and because deep research relies on retrieved evidence, report-wide claim verification is also necessary. To address these issues, we propose DEER, a benchmark for evaluating expert-level deep research reports. DEER systematizes evaluation criteria with an expert-developed taxonomy (7 dimensions, 25 subdimensions) operationalized as 101 fine-grained rubric items. We also provide task-specific Expert Evaluation Guidance to support LLM-based judging. Alongside rubric-based assessment, we propose a claim verification architecture that verifies both cited and uncited claims and quantifies evidence quality. Experiments show that while current deep research systems can produce structurally plausible reports that cite external evidence, there is room for improvement in fulfilling expert-level user requests and achieving logical completeness. Beyond simple performance comparisons, DEER makes system strengths and limitations interpretable and provides diagnostic signals for improvement. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_17776 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation Han, Janghoon Kim, Heegyu Lee, Changho Lee, Dahm Park, Min Hyung Song, Hosung Choi, Stanley Jungkyu Lee, Moontae Lee, Honglak Computation and Language Recent advances in large language models have enabled deep research systems that generate expert-level reports through multi-step reasoning and evidence-based synthesis. However, evaluating such reports remains challenging: report quality is multifaceted, making it difficult to determine what to assess and by what criteria; LLM-based judges may miss errors that require domain expertise to identify; and because deep research relies on retrieved evidence, report-wide claim verification is also necessary. To address these issues, we propose DEER, a benchmark for evaluating expert-level deep research reports. DEER systematizes evaluation criteria with an expert-developed taxonomy (7 dimensions, 25 subdimensions) operationalized as 101 fine-grained rubric items. We also provide task-specific Expert Evaluation Guidance to support LLM-based judging. Alongside rubric-based assessment, we propose a claim verification architecture that verifies both cited and uncited claims and quantifies evidence quality. Experiments show that while current deep research systems can produce structurally plausible reports that cite external evidence, there is room for improvement in fulfilling expert-level user requests and achieving logical completeness. Beyond simple performance comparisons, DEER makes system strengths and limitations interpretable and provides diagnostic signals for improvement. |
| title | DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2512.17776 |