Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909673979904000 |
|---|---|
| author | Gou, Boyu Huang, Zanming Ning, Yuting Gu, Yu Lin, Michael Qi, Weijian Kopanev, Andrei Yu, Botao Gutiérrez, Bernal Jiménez Shu, Yiheng Song, Chan Hee Wu, Jiaman Chen, Shijie Moussa, Hanane Nour Zhang, Tianshu Xie, Jian Li, Yifei Xue, Tianci Liao, Zeyi Zhang, Kai Zheng, Boyuan Cai, Zhaowei Rozgic, Viktor Ziyadi, Morteza Sun, Huan Su, Yu |
| author_facet | Gou, Boyu Huang, Zanming Ning, Yuting Gu, Yu Lin, Michael Qi, Weijian Kopanev, Andrei Yu, Botao Gutiérrez, Bernal Jiménez Shu, Yiheng Song, Chan Hee Wu, Jiaman Chen, Shijie Moussa, Hanane Nour Zhang, Tianshu Xie, Jian Li, Yifei Xue, Tianci Liao, Zeyi Zhang, Kai Zheng, Boyuan Cai, Zhaowei Rozgic, Viktor Ziyadi, Morteza Sun, Huan Su, Yu |
| contents | Agentic search such as Deep Research systems-where agents autonomously browse the web, synthesize information, and return comprehensive citation-backed answers-represents a major shift in how users interact with web-scale information. While promising greater efficiency and cognitive offloading, the growing complexity and open-endedness of agentic search have outpaced existing evaluation benchmarks and methodologies, which largely assume short search horizons and static answers. In this paper, we introduce Mind2Web 2, a benchmark of 130 realistic, high-quality, and long-horizon tasks that require real-time web browsing and extensive information synthesis, constructed with over 1000 hours of human labor. To address the challenge of evaluating time-varying and complex answers, we propose a novel Agent-as-a-Judge framework. Our method constructs task-specific judge agents based on a tree-structured rubric design to automatically assess both answer correctness and source attribution. We conduct a comprehensive evaluation of ten frontier agentic search systems and human performance, along with a detailed error analysis to draw insights for future development. The best-performing system, OpenAI Deep Research, can already achieve 50-70% of human performance while spending half the time, highlighting its great potential. Altogether, Mind2Web 2 provides a rigorous foundation for developing and benchmarking the next generation of agentic search systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_21506 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge Gou, Boyu Huang, Zanming Ning, Yuting Gu, Yu Lin, Michael Qi, Weijian Kopanev, Andrei Yu, Botao Gutiérrez, Bernal Jiménez Shu, Yiheng Song, Chan Hee Wu, Jiaman Chen, Shijie Moussa, Hanane Nour Zhang, Tianshu Xie, Jian Li, Yifei Xue, Tianci Liao, Zeyi Zhang, Kai Zheng, Boyuan Cai, Zhaowei Rozgic, Viktor Ziyadi, Morteza Sun, Huan Su, Yu Artificial Intelligence Computation and Language Agentic search such as Deep Research systems-where agents autonomously browse the web, synthesize information, and return comprehensive citation-backed answers-represents a major shift in how users interact with web-scale information. While promising greater efficiency and cognitive offloading, the growing complexity and open-endedness of agentic search have outpaced existing evaluation benchmarks and methodologies, which largely assume short search horizons and static answers. In this paper, we introduce Mind2Web 2, a benchmark of 130 realistic, high-quality, and long-horizon tasks that require real-time web browsing and extensive information synthesis, constructed with over 1000 hours of human labor. To address the challenge of evaluating time-varying and complex answers, we propose a novel Agent-as-a-Judge framework. Our method constructs task-specific judge agents based on a tree-structured rubric design to automatically assess both answer correctness and source attribution. We conduct a comprehensive evaluation of ten frontier agentic search systems and human performance, along with a detailed error analysis to draw insights for future development. The best-performing system, OpenAI Deep Research, can already achieve 50-70% of human performance while spending half the time, highlighting its great potential. Altogether, Mind2Web 2 provides a rigorous foundation for developing and benchmarking the next generation of agentic search systems. |
| title | Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2506.21506 |