Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2602.21143 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908851234668544 |
|---|---|
| author | Paul, Debjit Murphy, Daniel Gritta, Milan Cardenas, Ronald Prokhorov, Victor Bolliger, Lena Sophia Toker, Aysim Miles, Roy Oncescu, Andreea-Maria Sivakumar, Jasivan Alex Borchert, Philipp Elezi, Ismail Zhang, Meiru Lee, Ka Yiu Zhang, Guchun Wang, Jun Lampouras, Gerasimos |
| author_facet | Paul, Debjit Murphy, Daniel Gritta, Milan Cardenas, Ronald Prokhorov, Victor Bolliger, Lena Sophia Toker, Aysim Miles, Roy Oncescu, Andreea-Maria Sivakumar, Jasivan Alex Borchert, Philipp Elezi, Ismail Zhang, Meiru Lee, Ka Yiu Zhang, Guchun Wang, Jun Lampouras, Gerasimos |
| contents | Large language model (LLM)-based agents are increasingly used to solve complex tasks involving tool use, such as web browsing, code execution, and data analysis. However, current evaluation benchmarks do not adequately assess their ability to solve real-world tasks that require synthesizing information from multiple sources and inferring insights beyond simple fact retrieval. To address this, we introduce DEEPSYNTH, a novel benchmark designed to evaluate agents on realistic, time-consuming problems that combine information gathering, synthesis, and structured reasoning to produce insights. DEEPSYNTH contains 120 tasks collected across 7 domains and data sources covering 67 countries. DEEPSYNTH is constructed using a multi-stage data collection pipeline that requires annotators to collect official data sources, create hypotheses, perform manual analysis, and design tasks with verifiable answers. When evaluated on DEEPSYNTH, 11 state-of-the-art LLMs and deep research agents achieve a maximum F1 score of 8.97 and 17.5 on the LLM-judge metric, underscoring the difficulty of the benchmark. Our analysis reveals that current agents struggle with hallucinations and reasoning over large information spaces, highlighting DEEPSYNTH as a crucial benchmark for guiding future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_21143 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | A Benchmark for Deep Information Synthesis Paul, Debjit Murphy, Daniel Gritta, Milan Cardenas, Ronald Prokhorov, Victor Bolliger, Lena Sophia Toker, Aysim Miles, Roy Oncescu, Andreea-Maria Sivakumar, Jasivan Alex Borchert, Philipp Elezi, Ismail Zhang, Meiru Lee, Ka Yiu Zhang, Guchun Wang, Jun Lampouras, Gerasimos Artificial Intelligence Computation and Language Information Retrieval Machine Learning Large language model (LLM)-based agents are increasingly used to solve complex tasks involving tool use, such as web browsing, code execution, and data analysis. However, current evaluation benchmarks do not adequately assess their ability to solve real-world tasks that require synthesizing information from multiple sources and inferring insights beyond simple fact retrieval. To address this, we introduce DEEPSYNTH, a novel benchmark designed to evaluate agents on realistic, time-consuming problems that combine information gathering, synthesis, and structured reasoning to produce insights. DEEPSYNTH contains 120 tasks collected across 7 domains and data sources covering 67 countries. DEEPSYNTH is constructed using a multi-stage data collection pipeline that requires annotators to collect official data sources, create hypotheses, perform manual analysis, and design tasks with verifiable answers. When evaluated on DEEPSYNTH, 11 state-of-the-art LLMs and deep research agents achieve a maximum F1 score of 8.97 and 17.5 on the LLM-judge metric, underscoring the difficulty of the benchmark. Our analysis reveals that current agents struggle with hallucinations and reasoning over large information spaces, highlighting DEEPSYNTH as a crucial benchmark for guiding future research. |
| title | A Benchmark for Deep Information Synthesis |
| topic | Artificial Intelligence Computation and Language Information Retrieval Machine Learning |
| url | https://arxiv.org/abs/2602.21143 |