TL-GRPO: Turn-Level RL for Reasoning-Guided Iterative Optimization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914275246735360 |
|---|---|
| author | Li, Peiji Li, Linyang Sun, Handa Mai, Wenjin Chen, Yongkang Li, Xiaozhe Shen, Yue Ma, Yichuan Sun, Yiliu Cao, Jiaxi He, Zhishu Wang, Bo Zheng, Xiaoqing Bi, Zhaori Qiu, Xipeng Guo, Qipeng Chen, Kai Lin, Dahua |
| author_facet | Li, Peiji Li, Linyang Sun, Handa Mai, Wenjin Chen, Yongkang Li, Xiaozhe Shen, Yue Ma, Yichuan Sun, Yiliu Cao, Jiaxi He, Zhishu Wang, Bo Zheng, Xiaoqing Bi, Zhaori Qiu, Xipeng Guo, Qipeng Chen, Kai Lin, Dahua |
| contents | Large language models have demonstrated strong reasoning capabilities in complex tasks through tool integration, which is typically framed as a Markov Decision Process and optimized with trajectory-level RL algorithms such as GRPO. However, a common class of reasoning tasks, iterative optimization, presents distinct challenges: the agent interacts with the same underlying environment state across turns, and the value of a trajectory is determined by the best turn-level reward rather than cumulative returns. Existing GRPO-based methods cannot perform fine-grained, turn-level optimization in such settings, while black-box optimization methods discard prior knowledge and reasoning capabilities. To address this gap, we propose Turn-Level GRPO (TL-GRPO), a lightweight RL algorithm that performs turn-level group sampling for fine-grained optimization. We evaluate TL-GRPO on analog circuit sizing (ACS), a challenging scientific optimization task requiring multiple simulations and domain expertise. Results show that TL-GRPO outperforms standard GRPO and Bayesian optimization methods across various specifications. Furthermore, our 30B model trained with TL-GRPO achieves state-of-the-art performance on ACS tasks under same simulation budget, demonstrating both strong generalization and practical utility. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_16480 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | TL-GRPO: Turn-Level RL for Reasoning-Guided Iterative Optimization Li, Peiji Li, Linyang Sun, Handa Mai, Wenjin Chen, Yongkang Li, Xiaozhe Shen, Yue Ma, Yichuan Sun, Yiliu Cao, Jiaxi He, Zhishu Wang, Bo Zheng, Xiaoqing Bi, Zhaori Qiu, Xipeng Guo, Qipeng Chen, Kai Lin, Dahua Computation and Language Large language models have demonstrated strong reasoning capabilities in complex tasks through tool integration, which is typically framed as a Markov Decision Process and optimized with trajectory-level RL algorithms such as GRPO. However, a common class of reasoning tasks, iterative optimization, presents distinct challenges: the agent interacts with the same underlying environment state across turns, and the value of a trajectory is determined by the best turn-level reward rather than cumulative returns. Existing GRPO-based methods cannot perform fine-grained, turn-level optimization in such settings, while black-box optimization methods discard prior knowledge and reasoning capabilities. To address this gap, we propose Turn-Level GRPO (TL-GRPO), a lightweight RL algorithm that performs turn-level group sampling for fine-grained optimization. We evaluate TL-GRPO on analog circuit sizing (ACS), a challenging scientific optimization task requiring multiple simulations and domain expertise. Results show that TL-GRPO outperforms standard GRPO and Bayesian optimization methods across various specifications. Furthermore, our 30B model trained with TL-GRPO achieves state-of-the-art performance on ACS tasks under same simulation budget, demonstrating both strong generalization and practical utility. |
| title | TL-GRPO: Turn-Level RL for Reasoning-Guided Iterative Optimization |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.16480 |