Process-Supervised Reinforcement Learning for Interactive Multimodal Tool-Use Agents
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908545285357568 |
|---|---|
| author | Tan, Weiting Qu, Xinghua Tu, Ming Ge, Meng Liu, Andy T. Koehn, Philipp Lu, Lu |
| author_facet | Tan, Weiting Qu, Xinghua Tu, Ming Ge, Meng Liu, Andy T. Koehn, Philipp Lu, Lu |
| contents | Effective interactive tool use requires agents to master Tool Integrated Reasoning (TIR): a complex process involving multi-turn planning and long-context dialogue management. To train agents for this dynamic process, particularly in multi-modal contexts, we introduce a sandbox environment for reinforcement learning (RL) that supports interleaved speech-text rollouts. Our core strategy, Turn-level Adjudicated Reinforcement Learning (TARL), addresses the challenge of credit assignment in long-horizon tasks by employing a Large Language Model (LLM) as a judge to provide turn-level evaluation. To enhance exploration, we integrate a mixed-task training curriculum with mathematical reasoning problems. This unified approach boosts the task pass rate on the text-based $τ$-bench by over 6% compared to strong RL baselines. Crucially, we demonstrate our framework's suitability for fine-tuning a multi-modal foundation model for agentic tasks. By training a base multi-modal LLM on interleaved speech-text rollouts, we equip it with tool-use abilities, paving the way for more natural, voice-driven interactive agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_14480 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Process-Supervised Reinforcement Learning for Interactive Multimodal Tool-Use Agents Tan, Weiting Qu, Xinghua Tu, Ming Ge, Meng Liu, Andy T. Koehn, Philipp Lu, Lu Computation and Language Artificial Intelligence Multiagent Systems Effective interactive tool use requires agents to master Tool Integrated Reasoning (TIR): a complex process involving multi-turn planning and long-context dialogue management. To train agents for this dynamic process, particularly in multi-modal contexts, we introduce a sandbox environment for reinforcement learning (RL) that supports interleaved speech-text rollouts. Our core strategy, Turn-level Adjudicated Reinforcement Learning (TARL), addresses the challenge of credit assignment in long-horizon tasks by employing a Large Language Model (LLM) as a judge to provide turn-level evaluation. To enhance exploration, we integrate a mixed-task training curriculum with mathematical reasoning problems. This unified approach boosts the task pass rate on the text-based $τ$-bench by over 6% compared to strong RL baselines. Crucially, we demonstrate our framework's suitability for fine-tuning a multi-modal foundation model for agentic tasks. By training a base multi-modal LLM on interleaved speech-text rollouts, we equip it with tool-use abilities, paving the way for more natural, voice-driven interactive agents. |
| title | Process-Supervised Reinforcement Learning for Interactive Multimodal Tool-Use Agents |
| topic | Computation and Language Artificial Intelligence Multiagent Systems |
| url | https://arxiv.org/abs/2509.14480 |