Efficient Reinforcement Learning with Semantic and Token Entropy for LLM Reasoning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914257995563008 |
|---|---|
| author | Cao, Hongye Bai, Zhixin Peng, Ziyue Wang, Boyan Yang, Tianpei Huo, Jing Zhang, Yuyao Gao, Yang |
| author_facet | Cao, Hongye Bai, Zhixin Peng, Ziyue Wang, Boyan Yang, Tianpei Huo, Jing Zhang, Yuyao Gao, Yang |
| contents | Reinforcement learning with verifiable rewards (RLVR) has demonstrated superior performance in enhancing the reasoning capability of large language models (LLMs). However, this accuracy-oriented learning paradigm often suffers from entropy collapse, which reduces policy exploration and limits reasoning capabilities. To address this challenge, we propose an efficient reinforcement learning framework that leverages entropy signals at both the semantic and token levels to improve reasoning. From the data perspective, we introduce semantic entropy-guided curriculum learning, organizing training data from low to high semantic entropy to guide progressive optimization from easier to more challenging tasks. For the algorithmic design, we adopt non-uniform token treatment by imposing KL regularization on low-entropy tokens that critically impact policy exploration and applying stronger constraints on high-covariance portions within these tokens. By jointly optimizing data organization and algorithmic design, our method effectively mitigates entropy collapse and enhances LLM reasoning. Experimental results across 6 benchmarks with 3 different parameter-scale base models demonstrate that our method outperforms other entropy-based approaches in improving reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_04359 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Efficient Reinforcement Learning with Semantic and Token Entropy for LLM Reasoning Cao, Hongye Bai, Zhixin Peng, Ziyue Wang, Boyan Yang, Tianpei Huo, Jing Zhang, Yuyao Gao, Yang Artificial Intelligence Reinforcement learning with verifiable rewards (RLVR) has demonstrated superior performance in enhancing the reasoning capability of large language models (LLMs). However, this accuracy-oriented learning paradigm often suffers from entropy collapse, which reduces policy exploration and limits reasoning capabilities. To address this challenge, we propose an efficient reinforcement learning framework that leverages entropy signals at both the semantic and token levels to improve reasoning. From the data perspective, we introduce semantic entropy-guided curriculum learning, organizing training data from low to high semantic entropy to guide progressive optimization from easier to more challenging tasks. For the algorithmic design, we adopt non-uniform token treatment by imposing KL regularization on low-entropy tokens that critically impact policy exploration and applying stronger constraints on high-covariance portions within these tokens. By jointly optimizing data organization and algorithmic design, our method effectively mitigates entropy collapse and enhances LLM reasoning. Experimental results across 6 benchmarks with 3 different parameter-scale base models demonstrate that our method outperforms other entropy-based approaches in improving reasoning. |
| title | Efficient Reinforcement Learning with Semantic and Token Entropy for LLM Reasoning |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2512.04359 |