Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.19895 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917288769224704 |
|---|---|
| author | Wan, Zhongwei Shen, Yun Dou, Zhihao Zhou, Donghao Zhang, Yu Wang, Xin Shen, Hui Xiong, Jing Tao, Chaofan Zhong, Zixuan Huang, Peizhou Zhang, Mi |
| author_facet | Wan, Zhongwei Shen, Yun Dou, Zhihao Zhou, Donghao Zhang, Yu Wang, Xin Shen, Hui Xiong, Jing Tao, Chaofan Zhong, Zixuan Huang, Peizhou Zhang, Mi |
| contents | Reinforcement learning with verifiers (RLVR) is a central paradigm for improving large language model (LLM) reasoning, yet existing methods often suffer from limited exploration. Policies tend to collapse onto a few reasoning patterns and prematurely stop deep exploration, while conventional entropy regularization introduces only local stochasticity and fails to induce meaningful path-level diversity, leading to weak and unstable learning signals in group-based policy optimization. We propose DSDR, a Dual-Scale Diversity Regularization reinforcement learning framework that decomposes diversity in LLM reasoning into global and coupling components. Globally, DSDR promotes diversity among correct reasoning trajectories to explore distinct solution modes. Locally, it applies a length-invariant, token-level entropy regularization restricted to correct trajectories, preventing entropy collapse within each mode while preserving correctness. The two scales are coupled through a global-to-local allocation mechanism that emphasizes local regularization for more distinctive correct trajectories. We provide theoretical support showing that DSDR preserves optimal correctness under bounded regularization, sustains informative learning signals in group-based optimization, and yields a principled global-to-local coupling rule. Experiments on multiple reasoning benchmarks demonstrate consistent improvements in accuracy and pass@k, highlighting the importance of dual-scale diversity for deep exploration in RLVR. Code is available at https://github.com/SUSTechBruce/DSDR. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_19895 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning Wan, Zhongwei Shen, Yun Dou, Zhihao Zhou, Donghao Zhang, Yu Wang, Xin Shen, Hui Xiong, Jing Tao, Chaofan Zhong, Zixuan Huang, Peizhou Zhang, Mi Machine Learning Computation and Language Reinforcement learning with verifiers (RLVR) is a central paradigm for improving large language model (LLM) reasoning, yet existing methods often suffer from limited exploration. Policies tend to collapse onto a few reasoning patterns and prematurely stop deep exploration, while conventional entropy regularization introduces only local stochasticity and fails to induce meaningful path-level diversity, leading to weak and unstable learning signals in group-based policy optimization. We propose DSDR, a Dual-Scale Diversity Regularization reinforcement learning framework that decomposes diversity in LLM reasoning into global and coupling components. Globally, DSDR promotes diversity among correct reasoning trajectories to explore distinct solution modes. Locally, it applies a length-invariant, token-level entropy regularization restricted to correct trajectories, preventing entropy collapse within each mode while preserving correctness. The two scales are coupled through a global-to-local allocation mechanism that emphasizes local regularization for more distinctive correct trajectories. We provide theoretical support showing that DSDR preserves optimal correctness under bounded regularization, sustains informative learning signals in group-based optimization, and yields a principled global-to-local coupling rule. Experiments on multiple reasoning benchmarks demonstrate consistent improvements in accuracy and pass@k, highlighting the importance of dual-scale diversity for deep exploration in RLVR. Code is available at https://github.com/SUSTechBruce/DSDR. |
| title | DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2602.19895 |