Rethinking Stateful Tool Use in Multi-Turn Dialogues: Benchmarks and Challenges
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909615987359744 |
|---|---|
| author | Wang, Hongru Huang, Wenyu Wang, Yufei Xi, Yuanhao Lu, Jianqiao Zhang, Huan Hu, Nan Liu, Zeming Pan, Jeff Z. Wong, Kam-Fai |
| author_facet | Wang, Hongru Huang, Wenyu Wang, Yufei Xi, Yuanhao Lu, Jianqiao Zhang, Huan Hu, Nan Liu, Zeming Pan, Jeff Z. Wong, Kam-Fai |
| contents | Existing benchmarks that assess Language Models (LMs) as Language Agents (LAs) for tool use primarily focus on stateless, single-turn interactions or partial evaluations, such as tool selection in a single turn, overlooking the inherent stateful nature of interactions in multi-turn applications. To fulfill this gap, we propose \texttt{DialogTool}, a multi-turn dialogue dataset with stateful tool interactions considering the whole life cycle of tool use, across six key tasks in three stages: 1) \textit{tool creation}; 2) \textit{tool utilization}: tool awareness, tool selection, tool execution; and 3) \textit{role-consistent response}: response generation and role play. Furthermore, we build \texttt{VirtualMobile} -- an embodied virtual mobile evaluation environment to simulate API calls and assess the robustness of the created APIs\footnote{We will use tools and APIs alternatively, there are no significant differences between them in this paper.}. Taking advantage of these artifacts, we conduct comprehensive evaluation on 13 distinct open- and closed-source LLMs and provide detailed analysis at each stage, revealing that the existing state-of-the-art LLMs still cannot perform well to use tools over long horizons. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_13328 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Rethinking Stateful Tool Use in Multi-Turn Dialogues: Benchmarks and Challenges Wang, Hongru Huang, Wenyu Wang, Yufei Xi, Yuanhao Lu, Jianqiao Zhang, Huan Hu, Nan Liu, Zeming Pan, Jeff Z. Wong, Kam-Fai Computation and Language Existing benchmarks that assess Language Models (LMs) as Language Agents (LAs) for tool use primarily focus on stateless, single-turn interactions or partial evaluations, such as tool selection in a single turn, overlooking the inherent stateful nature of interactions in multi-turn applications. To fulfill this gap, we propose \texttt{DialogTool}, a multi-turn dialogue dataset with stateful tool interactions considering the whole life cycle of tool use, across six key tasks in three stages: 1) \textit{tool creation}; 2) \textit{tool utilization}: tool awareness, tool selection, tool execution; and 3) \textit{role-consistent response}: response generation and role play. Furthermore, we build \texttt{VirtualMobile} -- an embodied virtual mobile evaluation environment to simulate API calls and assess the robustness of the created APIs\footnote{We will use tools and APIs alternatively, there are no significant differences between them in this paper.}. Taking advantage of these artifacts, we conduct comprehensive evaluation on 13 distinct open- and closed-source LLMs and provide detailed analysis at each stage, revealing that the existing state-of-the-art LLMs still cannot perform well to use tools over long horizons. |
| title | Rethinking Stateful Tool Use in Multi-Turn Dialogues: Benchmarks and Challenges |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.13328 |