Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.13902 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915938754887680 |
|---|---|
| author | Li, Xiaofan Yang, Ming Ma, Zhiyuan Ma, Shichao Du, Jintao Cheng, Yu Wang, Weiqiang Zhang, Zhizhong Tan, Xin Qu, Yanyun Ma, Lizhuang Xie, Yuan |
| author_facet | Li, Xiaofan Yang, Ming Ma, Zhiyuan Ma, Shichao Du, Jintao Cheng, Yu Wang, Weiqiang Zhang, Zhizhong Tan, Xin Qu, Yanyun Ma, Lizhuang Xie, Yuan |
| contents | Reinforcement Learning with Verifiable Rewards (RLVR) has catalyzed significant advances in the reasoning capabilities of Large Language Models (LLMs). However, effectively managing the exploration and exploitation trade-off remains a critical challenge. In this paper, we fully analyze the exploration and exploitation dilemma of extremely hard and easy samples during the training and propose a new fine-grained trade-off mechanism. Concretely, we introduce a perplexity space disentangling strategy that divides the sample space into distinct exploration (high perplexity) and exploitation (low perplexity) subspaces, thereby mining fine-grained samples requiring exploration-exploitation trade-off. Subsequently, we propose a bidirectional reward allocation mechanism with a minimum impact on verification rewards to implement perplexity-guided exploration and exploitation, enabling more stable policy optimization. Finally, we have evaluated our method on two mainstream tasks: mathematical reasoning and function calling, and experimental results demonstrate the superiority of the proposed method, confirming its effectiveness in enhancing LLM performance by fine-grained exploration-exploitation trade-off. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_13902 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | DiPO: Disentangled Perplexity Policy Optimization for Fine-grained Exploration-Exploitation Trade-Off Li, Xiaofan Yang, Ming Ma, Zhiyuan Ma, Shichao Du, Jintao Cheng, Yu Wang, Weiqiang Zhang, Zhizhong Tan, Xin Qu, Yanyun Ma, Lizhuang Xie, Yuan Machine Learning Reinforcement Learning with Verifiable Rewards (RLVR) has catalyzed significant advances in the reasoning capabilities of Large Language Models (LLMs). However, effectively managing the exploration and exploitation trade-off remains a critical challenge. In this paper, we fully analyze the exploration and exploitation dilemma of extremely hard and easy samples during the training and propose a new fine-grained trade-off mechanism. Concretely, we introduce a perplexity space disentangling strategy that divides the sample space into distinct exploration (high perplexity) and exploitation (low perplexity) subspaces, thereby mining fine-grained samples requiring exploration-exploitation trade-off. Subsequently, we propose a bidirectional reward allocation mechanism with a minimum impact on verification rewards to implement perplexity-guided exploration and exploitation, enabling more stable policy optimization. Finally, we have evaluated our method on two mainstream tasks: mathematical reasoning and function calling, and experimental results demonstrate the superiority of the proposed method, confirming its effectiveness in enhancing LLM performance by fine-grained exploration-exploitation trade-off. |
| title | DiPO: Disentangled Perplexity Policy Optimization for Fine-grained Exploration-Exploitation Trade-Off |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2604.13902 |