Hista and Numca: Estimate State Value Effectively for LLM Reinforcement Learning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911727468150784 |
|---|---|
| author | Chen, Zizhe Dong, Jiqian Tian, Yizhou Yang, Garry Chen, Yongqiang Chen, Zhitang Cheng, James |
| author_facet | Chen, Zizhe Dong, Jiqian Tian, Yizhou Yang, Garry Chen, Yongqiang Chen, Zhitang Cheng, James |
| contents | Reinforcement learning (RL) refines large language models (LLMs) by directly optimizing model behavior through reward signals. While accurate state value estimation is critical for stable training in classical RL, it remains an underexplored challenge in LLM post-training. In this work, we introduce the State Value Estimation Benchmark (SVEB) to assess state estimation within existing RL frameworks and show that critics in standard approaches like PPO collapse to a coarse group-average baseline. To address this, we propose two techniques: Numca, which leverages numerical spans as gradable milestones for state value estimation, and Hista, a framework that uses LLM's hidden states as representation to weighted average disjoint rollouts and their return. Extensive experiments demonstrate that both methods yield more accurate state value estimates and enhance training performance across different RL algorithms and model sizes without incurring significant computational overhead. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_29782 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Hista and Numca: Estimate State Value Effectively for LLM Reinforcement Learning Chen, Zizhe Dong, Jiqian Tian, Yizhou Yang, Garry Chen, Yongqiang Chen, Zhitang Cheng, James Machine Learning Artificial Intelligence Computation and Language I.2.7 Reinforcement learning (RL) refines large language models (LLMs) by directly optimizing model behavior through reward signals. While accurate state value estimation is critical for stable training in classical RL, it remains an underexplored challenge in LLM post-training. In this work, we introduce the State Value Estimation Benchmark (SVEB) to assess state estimation within existing RL frameworks and show that critics in standard approaches like PPO collapse to a coarse group-average baseline. To address this, we propose two techniques: Numca, which leverages numerical spans as gradable milestones for state value estimation, and Hista, a framework that uses LLM's hidden states as representation to weighted average disjoint rollouts and their return. Extensive experiments demonstrate that both methods yield more accurate state value estimates and enhance training performance across different RL algorithms and model sizes without incurring significant computational overhead. |
| title | Hista and Numca: Estimate State Value Effectively for LLM Reinforcement Learning |
| topic | Machine Learning Artificial Intelligence Computation and Language I.2.7 |
| url | https://arxiv.org/abs/2605.29782 |