CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908533479440384 |
|---|---|
| author | Dai, Runpeng Song, Linfeng Liu, Haolin Liang, Zhenwen Yu, Dian Mi, Haitao Tu, Zhaopeng Liu, Rui Zheng, Tong Zhu, Hongtu Yu, Dong |
| author_facet | Dai, Runpeng Song, Linfeng Liu, Haolin Liang, Zhenwen Yu, Dian Mi, Haitao Tu, Zhaopeng Liu, Rui Zheng, Tong Zhu, Hongtu Yu, Dong |
| contents | Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for enhancing the reasoning ability of Large Language Models (LLMs). Yet current RLVR methods often explore poorly, leading to premature convergence and entropy collapse. To address this challenge, we introduce Curiosity-Driven Exploration (CDE), a framework that leverages the model's own intrinsic sense of curiosity to guide exploration. We formalize curiosity with signals from both the actor and the critic: for the actor, we use perplexity over its generated response, and for the critic, we use the variance of value estimates from a multi-head architecture. Both signals serve as an exploration bonus within the RLVR framework to guide the model. Our theoretical analysis shows that the actor-wise bonus inherently penalizes overconfident errors and promotes diversity among correct responses; moreover, we connect the critic-wise bonus to the well-established count-based exploration bonus in RL. Empirically, our method achieves an approximate +3 point improvement over standard RLVR using GRPO/PPO on AIME benchmarks. Further analysis identifies a calibration collapse mechanism within RLVR, shedding light on common LLM failure modes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_09675 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models Dai, Runpeng Song, Linfeng Liu, Haolin Liang, Zhenwen Yu, Dian Mi, Haitao Tu, Zhaopeng Liu, Rui Zheng, Tong Zhu, Hongtu Yu, Dong Computation and Language Artificial Intelligence Machine Learning Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for enhancing the reasoning ability of Large Language Models (LLMs). Yet current RLVR methods often explore poorly, leading to premature convergence and entropy collapse. To address this challenge, we introduce Curiosity-Driven Exploration (CDE), a framework that leverages the model's own intrinsic sense of curiosity to guide exploration. We formalize curiosity with signals from both the actor and the critic: for the actor, we use perplexity over its generated response, and for the critic, we use the variance of value estimates from a multi-head architecture. Both signals serve as an exploration bonus within the RLVR framework to guide the model. Our theoretical analysis shows that the actor-wise bonus inherently penalizes overconfident errors and promotes diversity among correct responses; moreover, we connect the critic-wise bonus to the well-established count-based exploration bonus in RL. Empirically, our method achieves an approximate +3 point improvement over standard RLVR using GRPO/PPO on AIME benchmarks. Further analysis identifies a calibration collapse mechanism within RLVR, shedding light on common LLM failure modes. |
| title | CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2509.09675 |