AutoTool: Automatic Scaling of Tool-Use Capabilities in RL via Decoupled Entropy Constraints
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917340610822144 |
|---|---|
| author | Zeng, Yirong Ding, Xiao Liu, Yufei Wang, Yuxian Du, Qunyao Hou, Yutai Ning, Wu Song, Haonan Tang, Duyu Tu, Dandan Qin, Bing Liu, Ting |
| author_facet | Zeng, Yirong Ding, Xiao Liu, Yufei Wang, Yuxian Du, Qunyao Hou, Yutai Ning, Wu Song, Haonan Tang, Duyu Tu, Dandan Qin, Bing Liu, Ting |
| contents | Tool use represents a critical capability for AI agents, with recent advances focusing on leveraging reinforcement learning (RL) to scale up the explicit reasoning process to achieve better performance. However, there are some key challenges for tool use in current RL-based scaling approaches: (a) direct RL training often struggles to scale up thinking length sufficiently to solve complex problems, and (b) scaled-up models tend to overthink simpler problems, resulting in substantial token inefficiency. To address these challenges, we propose a novel training paradigm that first employs warm-up supervised fine-tuning to help models distinguish between simple and complex problems, followed by RL that enable models to automatically determine appropriate reasoning trajectories. Furthermore, to tackle the issue of automatic thinking-length scaling, we discover that entropy-based optimization objectives effectively maintain model diversity while successfully unlocking the model's scaling capabilities. Based on this insight, we introduce an entropy-based long-short reasoning fusion RL strategy. Our experiments on three benchmarks demonstrate that model successfully achieves auto-scaling for efficient tool use, achieving significant 9.8\% accuracy improvements while reducing computational overhead by \textasciitilde81\%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_13348 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | AutoTool: Automatic Scaling of Tool-Use Capabilities in RL via Decoupled Entropy Constraints Zeng, Yirong Ding, Xiao Liu, Yufei Wang, Yuxian Du, Qunyao Hou, Yutai Ning, Wu Song, Haonan Tang, Duyu Tu, Dandan Qin, Bing Liu, Ting Artificial Intelligence Tool use represents a critical capability for AI agents, with recent advances focusing on leveraging reinforcement learning (RL) to scale up the explicit reasoning process to achieve better performance. However, there are some key challenges for tool use in current RL-based scaling approaches: (a) direct RL training often struggles to scale up thinking length sufficiently to solve complex problems, and (b) scaled-up models tend to overthink simpler problems, resulting in substantial token inefficiency. To address these challenges, we propose a novel training paradigm that first employs warm-up supervised fine-tuning to help models distinguish between simple and complex problems, followed by RL that enable models to automatically determine appropriate reasoning trajectories. Furthermore, to tackle the issue of automatic thinking-length scaling, we discover that entropy-based optimization objectives effectively maintain model diversity while successfully unlocking the model's scaling capabilities. Based on this insight, we introduce an entropy-based long-short reasoning fusion RL strategy. Our experiments on three benchmarks demonstrate that model successfully achieves auto-scaling for efficient tool use, achieving significant 9.8\% accuracy improvements while reducing computational overhead by \textasciitilde81\%. |
| title | AutoTool: Automatic Scaling of Tool-Use Capabilities in RL via Decoupled Entropy Constraints |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2603.13348 |