Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2601.22776 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918429386080256 |
|---|---|
| author | Ma, Shichao Ma, Zhiyuan Yang, Ming Li, Xiaofan Wu, Xing Du, Jintao Cheng, Yu Wang, Weiqiang Liu, Qiliang Zhou, Zhengyang Wang, Yang |
| author_facet | Ma, Shichao Ma, Zhiyuan Yang, Ming Li, Xiaofan Wu, Xing Du, Jintao Cheng, Yu Wang, Weiqiang Liu, Qiliang Zhou, Zhengyang Wang, Yang |
| contents | Multi-turn tool-integrated reasoning enables Large Language Models (LLMs) to solve complex tasks through iterative information retrieval. However, current reinforcement learning (RL) frameworks for search-augmented reasoning predominantly rely on sparse outcome-level rewards, leading to a "Double Homogenization Dilemma." This manifests as (1) Process homogenization, where the thinking, reasoning, and tooling involved in generation are ignored. (2) Intra-group homogenization, coarse-grained outcome rewards often lead to inefficiencies in intra-group advantage estimation with methods like Group Relative Policy Optimization (GRPO) during sampling. To address this, we propose Turn-level Stage-aware Policy Optimization (TSPO). TSPO introduces the First-Occurrence Latent Reward (FOLR) mechanism, allocating partial rewards to the step where the ground-truth answer first appears, thereby preserving process-level signals and increasing reward variance within groups without requiring external reward models or any annotations. Extensive experiments demonstrate that TSPO significantly outperforms state-of-the-art baselines, achieving average performance gains of 24% and 13.6% on Qwen2.5-3B and 7B models, respectively. Code is available at https://github.com/Flipped-May/TSPO. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_22776 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | TSPO: Breaking the Double Homogenization Dilemma in Multi-turn Search Policy Optimization Ma, Shichao Ma, Zhiyuan Yang, Ming Li, Xiaofan Wu, Xing Du, Jintao Cheng, Yu Wang, Weiqiang Liu, Qiliang Zhou, Zhengyang Wang, Yang Artificial Intelligence Multi-turn tool-integrated reasoning enables Large Language Models (LLMs) to solve complex tasks through iterative information retrieval. However, current reinforcement learning (RL) frameworks for search-augmented reasoning predominantly rely on sparse outcome-level rewards, leading to a "Double Homogenization Dilemma." This manifests as (1) Process homogenization, where the thinking, reasoning, and tooling involved in generation are ignored. (2) Intra-group homogenization, coarse-grained outcome rewards often lead to inefficiencies in intra-group advantage estimation with methods like Group Relative Policy Optimization (GRPO) during sampling. To address this, we propose Turn-level Stage-aware Policy Optimization (TSPO). TSPO introduces the First-Occurrence Latent Reward (FOLR) mechanism, allocating partial rewards to the step where the ground-truth answer first appears, thereby preserving process-level signals and increasing reward variance within groups without requiring external reward models or any annotations. Extensive experiments demonstrate that TSPO significantly outperforms state-of-the-art baselines, achieving average performance gains of 24% and 13.6% on Qwen2.5-3B and 7B models, respectively. Code is available at https://github.com/Flipped-May/TSPO. |
| title | TSPO: Breaking the Double Homogenization Dilemma in Multi-turn Search Policy Optimization |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2601.22776 |