TRE: Encouraging Exploration in the Trust Region
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912873403383808 |
|---|---|
| author | Huang, Chao Lu, Yujing Li, Quangang Wang, Shenghe Wang, Yan Zhang, Yueyang Xia, Long Zhao, Jiashu Sun, Zhiyuan Shi, Daiting Liu, Tingwen |
| author_facet | Huang, Chao Lu, Yujing Li, Quangang Wang, Shenghe Wang, Yan Zhang, Yueyang Xia, Long Zhao, Jiashu Sun, Zhiyuan Shi, Daiting Liu, Tingwen |
| contents | Entropy regularization is a standard technique in reinforcement learning (RL) to enhance exploration, yet it yields negligible effects or even degrades performance in Large Language Models (LLMs). We attribute this failure to the cumulative tail risk inherent to LLMs with massive vocabularies and long generation horizons. In such environments, standard global entropy maximization indiscriminately dilutes probability mass into the vast tail of invalid tokens rather than focusing on plausible candidates, thereby disrupting coherent reasoning. To address this, we propose Trust Region Entropy (TRE), a method that encourages exploration strictly within the model's trust region. Extensive experiments across mathematical reasoning (MATH), combinatorial search (Countdown), and preference alignment (HH) tasks demonstrate that TRE consistently outperforms vanilla PPO, standard entropy regularization, and other exploration baselines. Our code is available at https://github.com/WhyChaos/TRE-Encouraging-Exploration-in-the-Trust-Region. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_03635 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | TRE: Encouraging Exploration in the Trust Region Huang, Chao Lu, Yujing Li, Quangang Wang, Shenghe Wang, Yan Zhang, Yueyang Xia, Long Zhao, Jiashu Sun, Zhiyuan Shi, Daiting Liu, Tingwen Computation and Language Machine Learning Entropy regularization is a standard technique in reinforcement learning (RL) to enhance exploration, yet it yields negligible effects or even degrades performance in Large Language Models (LLMs). We attribute this failure to the cumulative tail risk inherent to LLMs with massive vocabularies and long generation horizons. In such environments, standard global entropy maximization indiscriminately dilutes probability mass into the vast tail of invalid tokens rather than focusing on plausible candidates, thereby disrupting coherent reasoning. To address this, we propose Trust Region Entropy (TRE), a method that encourages exploration strictly within the model's trust region. Extensive experiments across mathematical reasoning (MATH), combinatorial search (Countdown), and preference alignment (HH) tasks demonstrate that TRE consistently outperforms vanilla PPO, standard entropy regularization, and other exploration baselines. Our code is available at https://github.com/WhyChaos/TRE-Encouraging-Exploration-in-the-Trust-Region. |
| title | TRE: Encouraging Exploration in the Trust Region |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2602.03635 |