Advancing and Benchmarking Personalized Tool Invocation for LLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910930128863232 |
|---|---|
| author | Huang, Xu Huang, Yuefeng Liu, Weiwen Zeng, Xingshan Wang, Yasheng Tang, Ruiming Xie, Hong Lian, Defu |
| author_facet | Huang, Xu Huang, Yuefeng Liu, Weiwen Zeng, Xingshan Wang, Yasheng Tang, Ruiming Xie, Hong Lian, Defu |
| contents | Tool invocation is a crucial mechanism for extending the capabilities of Large Language Models (LLMs) and has recently garnered significant attention. It enables LLMs to solve complex problems through tool calls while accessing up-to-date world knowledge. However, existing work primarily focuses on the fundamental ability of LLMs to invoke tools for problem-solving, without considering personalized constraints in tool invocation. In this work, we introduce the concept of Personalized Tool Invocation and define two key tasks: Tool Preference and Profile-dependent Query. Tool Preference addresses user preferences when selecting among functionally similar tools, while Profile-dependent Query considers cases where a user query lacks certain tool parameters, requiring the model to infer them from the user profile. To tackle these challenges, we propose PTool, a data synthesis framework designed for personalized tool invocation. Additionally, we construct \textbf{PTBench}, the first benchmark for evaluating personalized tool invocation. We then fine-tune various open-source models, demonstrating the effectiveness of our framework and providing valuable insights. Our benchmark is public at https://github.com/hyfshadow/PTBench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_04072 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Advancing and Benchmarking Personalized Tool Invocation for LLMs Huang, Xu Huang, Yuefeng Liu, Weiwen Zeng, Xingshan Wang, Yasheng Tang, Ruiming Xie, Hong Lian, Defu Computation and Language Artificial Intelligence Tool invocation is a crucial mechanism for extending the capabilities of Large Language Models (LLMs) and has recently garnered significant attention. It enables LLMs to solve complex problems through tool calls while accessing up-to-date world knowledge. However, existing work primarily focuses on the fundamental ability of LLMs to invoke tools for problem-solving, without considering personalized constraints in tool invocation. In this work, we introduce the concept of Personalized Tool Invocation and define two key tasks: Tool Preference and Profile-dependent Query. Tool Preference addresses user preferences when selecting among functionally similar tools, while Profile-dependent Query considers cases where a user query lacks certain tool parameters, requiring the model to infer them from the user profile. To tackle these challenges, we propose PTool, a data synthesis framework designed for personalized tool invocation. Additionally, we construct \textbf{PTBench}, the first benchmark for evaluating personalized tool invocation. We then fine-tune various open-source models, demonstrating the effectiveness of our framework and providing valuable insights. Our benchmark is public at https://github.com/hyfshadow/PTBench. |
| title | Advancing and Benchmarking Personalized Tool Invocation for LLMs |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2505.04072 |