Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.26114 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915524944855040 |
|---|---|
| author | Park, Jaesung R. Kim, Junsu Kim, Gyeongman Jo, Jinyoung Choi, Sean Cho, Jaewoong Ryu, Ernest K. |
| author_facet | Park, Jaesung R. Kim, Junsu Kim, Gyeongman Jo, Jinyoung Choi, Sean Cho, Jaewoong Ryu, Ernest K. |
| contents | Reinforcement learning with verifiable rewards (RLVR) has recently emerged as the leading approach for enhancing the reasoning capabilities of large language models (LLMs). However, RLVR is prone to entropy collapse, where the LLM quickly converges to a near-deterministic form, hindering exploration and progress during prolonged RL training. In this work, we reveal that the clipping mechanism in PPO and GRPO induces biases on entropy. Through theoretical and empirical analyses, we show that clip-low increases entropy, while clip-high decreases it. Further, under standard clipping parameters, the effect of clip-high dominates, resulting in an overall entropy reduction even when purely random rewards are provided to the RL algorithm. Our findings highlight an overlooked confounding factor in RLVR: independent of the reward signal, the clipping mechanism influences entropy, which in turn affects the reasoning behavior. Furthermore, our analysis demonstrates that clipping can be deliberately used to control entropy. Specifically, with a more aggressive clip-low value, one can increase entropy, promote exploration, and ultimately prevent entropy collapse in RLVR training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_26114 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models Park, Jaesung R. Kim, Junsu Kim, Gyeongman Jo, Jinyoung Choi, Sean Cho, Jaewoong Ryu, Ernest K. Machine Learning Reinforcement learning with verifiable rewards (RLVR) has recently emerged as the leading approach for enhancing the reasoning capabilities of large language models (LLMs). However, RLVR is prone to entropy collapse, where the LLM quickly converges to a near-deterministic form, hindering exploration and progress during prolonged RL training. In this work, we reveal that the clipping mechanism in PPO and GRPO induces biases on entropy. Through theoretical and empirical analyses, we show that clip-low increases entropy, while clip-high decreases it. Further, under standard clipping parameters, the effect of clip-high dominates, resulting in an overall entropy reduction even when purely random rewards are provided to the RL algorithm. Our findings highlight an overlooked confounding factor in RLVR: independent of the reward signal, the clipping mechanism influences entropy, which in turn affects the reasoning behavior. Furthermore, our analysis demonstrates that clipping can be deliberately used to control entropy. Specifically, with a more aggressive clip-low value, one can increase entropy, promote exploration, and ultimately prevent entropy collapse in RLVR training. |
| title | Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2509.26114 |