Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.24668 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915583327469568 |
|---|---|
| author | Deng, Mingyi Huang, Lijun Fan, Yani Zhang, Jiayi Ren, Fashen Bai, Jinyi Yang, Fuzhen Miao, Dayi Yu, Zhaoyang Wu, Yifan Zhang, Yanfei Teng, Fengwei Wan, Yingjia Hu, Song Li, Yude Jin, Xin Hu, Conghao Li, Haoyu Fu, Qirui Zhong, Tai Wang, Xinyu Tang, Xiangru Tang, Nan Wu, Chenglin Luo, Yuyu |
| author_facet | Deng, Mingyi Huang, Lijun Fan, Yani Zhang, Jiayi Ren, Fashen Bai, Jinyi Yang, Fuzhen Miao, Dayi Yu, Zhaoyang Wu, Yifan Zhang, Yanfei Teng, Fengwei Wan, Yingjia Hu, Song Li, Yude Jin, Xin Hu, Conghao Li, Haoyu Fu, Qirui Zhong, Tai Wang, Xinyu Tang, Xiangru Tang, Nan Wu, Chenglin Luo, Yuyu |
| contents | Language agents have demonstrated remarkable potential in web search and information retrieval. However, these search agents assume user queries are complete and unambiguous, an assumption that diverges from reality where users begin with incomplete queries requiring clarification through interaction. Yet most agents lack interactive mechanisms during the search process, and existing benchmarks cannot assess this capability. To address this gap, we introduce InteractComp, a benchmark designed to evaluate whether search agents can recognize query ambiguity and actively interact to resolve it during search. Following the principle of easy to verify, interact to disambiguate, we construct 210 expert-curated questions across 9 domains through a target-distractor methodology that creates genuine ambiguity resolvable only through interaction. Evaluation of 17 models reveals striking failure: the best model achieves only 13.73% accuracy despite 71.50% with complete context, exposing systematic overconfidence rather than reasoning deficits. Forced interaction produces dramatic gains, demonstrating latent capability current strategies fail to engage. Longitudinal analysis shows interaction capabilities stagnated over 15 months while search performance improved seven-fold, revealing a critical blind spot. This stagnation, coupled with the immediate feedback inherent to search tasks, makes InteractComp a valuable resource for both evaluating and training interaction capabilities in search agents. The code is available at https://github.com/FoundationAgents/InteractComp. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_24668 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | InteractComp: Evaluating Search Agents With Ambiguous Queries Deng, Mingyi Huang, Lijun Fan, Yani Zhang, Jiayi Ren, Fashen Bai, Jinyi Yang, Fuzhen Miao, Dayi Yu, Zhaoyang Wu, Yifan Zhang, Yanfei Teng, Fengwei Wan, Yingjia Hu, Song Li, Yude Jin, Xin Hu, Conghao Li, Haoyu Fu, Qirui Zhong, Tai Wang, Xinyu Tang, Xiangru Tang, Nan Wu, Chenglin Luo, Yuyu Computation and Language Artificial Intelligence Language agents have demonstrated remarkable potential in web search and information retrieval. However, these search agents assume user queries are complete and unambiguous, an assumption that diverges from reality where users begin with incomplete queries requiring clarification through interaction. Yet most agents lack interactive mechanisms during the search process, and existing benchmarks cannot assess this capability. To address this gap, we introduce InteractComp, a benchmark designed to evaluate whether search agents can recognize query ambiguity and actively interact to resolve it during search. Following the principle of easy to verify, interact to disambiguate, we construct 210 expert-curated questions across 9 domains through a target-distractor methodology that creates genuine ambiguity resolvable only through interaction. Evaluation of 17 models reveals striking failure: the best model achieves only 13.73% accuracy despite 71.50% with complete context, exposing systematic overconfidence rather than reasoning deficits. Forced interaction produces dramatic gains, demonstrating latent capability current strategies fail to engage. Longitudinal analysis shows interaction capabilities stagnated over 15 months while search performance improved seven-fold, revealing a critical blind spot. This stagnation, coupled with the immediate feedback inherent to search tasks, makes InteractComp a valuable resource for both evaluating and training interaction capabilities in search agents. The code is available at https://github.com/FoundationAgents/InteractComp. |
| title | InteractComp: Evaluating Search Agents With Ambiguous Queries |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.24668 |