Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918016983236608 |
|---|---|
| author | Zhang, Shaokun Dong, Yi Zhang, Jieyu Kautz, Jan Catanzaro, Bryan Tao, Andrew Wu, Qingyun Yu, Zhiding Liu, Guilin |
| author_facet | Zhang, Shaokun Dong, Yi Zhang, Jieyu Kautz, Jan Catanzaro, Bryan Tao, Andrew Wu, Qingyun Yu, Zhiding Liu, Guilin |
| contents | Enabling large language models with external tools has become a pivotal strategy for extending their functionality beyond text space. To enhance LLMs' tool-calling abilities, previous approaches primarily rely on supervised fine-tuning (SFT) with trajectories distilled from stronger models, often resulting in imitative reasoning that limits generalization. In this work, we explore rule-based reinforcement learning to enhance tool-calling in LLMs, resulting in Nemotron-Research-Tool-N1, a series of tool-calling reasoning models. Rather than enforcing supervision over intermediate distilled reasoning traces, Tool-N1 is trained with a binary RL reward that assesses only the format validity and functional correctness of tool invocations. This lightweight supervision allows the model to develop reasoning strategies independently, without relying on annotated trajectories. Experiments on several major benchmarks show that Tool-N1-7B/14B clearly outperform GPT-4o. We conduct a systematic study on the design of rule-based reinforcement learning strategies for training tool-calling models. Using 5,518 distilled reasoning trajectories, we compare SFT, RL, and the SFT-then-RL pipeline, finding that the widely adopted SFT-then-RL paradigm does not necessarily outperform pure RL. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_00024 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning Zhang, Shaokun Dong, Yi Zhang, Jieyu Kautz, Jan Catanzaro, Bryan Tao, Andrew Wu, Qingyun Yu, Zhiding Liu, Guilin Computation and Language Artificial Intelligence Enabling large language models with external tools has become a pivotal strategy for extending their functionality beyond text space. To enhance LLMs' tool-calling abilities, previous approaches primarily rely on supervised fine-tuning (SFT) with trajectories distilled from stronger models, often resulting in imitative reasoning that limits generalization. In this work, we explore rule-based reinforcement learning to enhance tool-calling in LLMs, resulting in Nemotron-Research-Tool-N1, a series of tool-calling reasoning models. Rather than enforcing supervision over intermediate distilled reasoning traces, Tool-N1 is trained with a binary RL reward that assesses only the format validity and functional correctness of tool invocations. This lightweight supervision allows the model to develop reasoning strategies independently, without relying on annotated trajectories. Experiments on several major benchmarks show that Tool-N1-7B/14B clearly outperform GPT-4o. We conduct a systematic study on the design of rule-based reinforcement learning strategies for training tool-calling models. Using 5,518 distilled reasoning trajectories, we compare SFT, RL, and the SFT-then-RL pipeline, finding that the widely adopted SFT-then-RL paradigm does not necessarily outperform pure RL. |
| title | Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2505.00024 |