The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911428646010880 |
|---|---|
| author | Li, Yingru Xu, Jiawei Li, Ziniu Liu, Jiacai Liu, Wei Tong, Yuxuan Zheng, Longtao Xue, Zhenghai Zhang, Yaxiang Cai, Tianle Zhang, Ge Liu, Qian Wang, Baoxiang |
| author_facet | Li, Yingru Xu, Jiawei Li, Ziniu Liu, Jiacai Liu, Wei Tong, Yuxuan Zheng, Longtao Xue, Zhenghai Zhang, Yaxiang Cai, Tianle Zhang, Ge Liu, Qian Wang, Baoxiang |
| contents | Reinforcement Learning (RL) for Large Language Models (LLMs) often suffers from training collapse in long-horizon tasks due to exploding gradient variance. To mitigate this, a baseline is commonly introduced for advantage computation; however, traditional value models remain difficult to optimize, and standard group-based baselines overlook sequence heterogeneity. Although classic optimal baseline theory can achieve global variance reduction, it neglects token heterogeneity and requires prohibitive gradient-based computation. In this work, we derive the Optimal Token Baseline (OTB) from first principles, proving that gradient updates should be weighted inversely to their cumulative gradient norm. To ensure efficiency, we propose the Logit-Gradient Proxy that approximates the gradient norm using only forward-pass probabilities. Our method achieves training stability and matches the performance of large group sizes ($N=32$) with only $N=4$, reducing token consumption by over 65% across single-turn and tool-integrated reasoning tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_07078 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL Li, Yingru Xu, Jiawei Li, Ziniu Liu, Jiacai Liu, Wei Tong, Yuxuan Zheng, Longtao Xue, Zhenghai Zhang, Yaxiang Cai, Tianle Zhang, Ge Liu, Qian Wang, Baoxiang Machine Learning Artificial Intelligence Reinforcement Learning (RL) for Large Language Models (LLMs) often suffers from training collapse in long-horizon tasks due to exploding gradient variance. To mitigate this, a baseline is commonly introduced for advantage computation; however, traditional value models remain difficult to optimize, and standard group-based baselines overlook sequence heterogeneity. Although classic optimal baseline theory can achieve global variance reduction, it neglects token heterogeneity and requires prohibitive gradient-based computation. In this work, we derive the Optimal Token Baseline (OTB) from first principles, proving that gradient updates should be weighted inversely to their cumulative gradient norm. To ensure efficiency, we propose the Logit-Gradient Proxy that approximates the gradient norm using only forward-pass probabilities. Our method achieves training stability and matches the performance of large group sizes ($N=32$) with only $N=4$, reducing token consumption by over 65% across single-turn and tool-integrated reasoning tasks. |
| title | The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2602.07078 |