Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2506.08125 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917414594150400 |
|---|---|
| author | Liu, Hanbing Cao, Lang Ren, Yuanyi Zhou, Mengyu Dong, Haoyu Ma, Xiaojun Han, Shi Zhang, Dongmei |
| author_facet | Liu, Hanbing Cao, Lang Ren, Yuanyi Zhou, Mengyu Dong, Haoyu Ma, Xiaojun Han, Shi Zhang, Dongmei |
| contents | Large language models (LLMs) show strong reasoning abilities but often produce unnecessarily long explanations that reduce efficiency. Although reinforcement learning (RL) has been used to improve reasoning, most methods focus on accuracy and rely on uniform length-based rewards that overlook the differing contributions of individual tokens, often harming correctness. We revisit length optimization in RL through the perspective of token significance. Observing that many chain-of-thought (CoT) tokens contribute little to the final answer, we introduce a significance-aware length reward that selectively penalizes insignificance tokens, reducing redundancy while preserving essential reasoning. We also propose a dynamic length reward that encourages more detailed reasoning early in training and gradually shifts toward conciseness as learning progresses. Integrating these components into standard policy optimization yields a framework that improves both reasoning efficiency and accuracy. Experiments across multiple benchmarks demonstrate substantial reductions in response length while preserving or improving correctness, highlighting the importance of modeling token significance for efficient LLM reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_08125 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learning Liu, Hanbing Cao, Lang Ren, Yuanyi Zhou, Mengyu Dong, Haoyu Ma, Xiaojun Han, Shi Zhang, Dongmei Machine Learning Computation and Language Large language models (LLMs) show strong reasoning abilities but often produce unnecessarily long explanations that reduce efficiency. Although reinforcement learning (RL) has been used to improve reasoning, most methods focus on accuracy and rely on uniform length-based rewards that overlook the differing contributions of individual tokens, often harming correctness. We revisit length optimization in RL through the perspective of token significance. Observing that many chain-of-thought (CoT) tokens contribute little to the final answer, we introduce a significance-aware length reward that selectively penalizes insignificance tokens, reducing redundancy while preserving essential reasoning. We also propose a dynamic length reward that encourages more detailed reasoning early in training and gradually shifts toward conciseness as learning progresses. Integrating these components into standard policy optimization yields a framework that improves both reasoning efficiency and accuracy. Experiments across multiple benchmarks demonstrate substantial reductions in response length while preserving or improving correctness, highlighting the importance of modeling token significance for efficient LLM reasoning. |
| title | Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learning |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2506.08125 |