Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.07809 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908978828541952 |
|---|---|
| author | Liu, Huanyu Li, Jia Dong, Yihong Yu, Chang Chen, Taozhi Wang, Lecheng Tao, Yongding Gu, Bin Li, Ge |
| author_facet | Liu, Huanyu Li, Jia Dong, Yihong Yu, Chang Chen, Taozhi Wang, Lecheng Tao, Yongding Gu, Bin Li, Ge |
| contents | Reinforcement learning with verifiable reward (RLVR) has become a promising paradigm for post-training large language models (LLMs) to improve their reasoning capability. However, when the rollout accuracy is low on hard problems, the reward becomes sparse, limiting learning efficiency and causing exploration bottlenecks. Existing approaches either rely on teacher models for distillation or filter out difficult problems, which limits scalability or restricts reasoning improvement through exploration.
We propose EvoCoT, a self-evolving curriculum learning framework based on two-stage chain-of-thought (CoT) reasoning optimization. EvoCoT constrains the exploration space by self-generating and verifying CoT trajectories, then gradually shortens CoT steps to expand the space in a controlled way. The framework enables LLMs to stably learn from initially unsolved hard problems under sparse rewards. We apply EvoCoT to multiple LLM families, including Qwen, DeepSeek, and Llama. Experiments show that EvoCoT enables LLMs to solve previously unsolved problems, improves reasoning capability without external CoT supervision, and is compatible with various RL fine-tuning methods. We release the source code to support future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_07809 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learning Liu, Huanyu Li, Jia Dong, Yihong Yu, Chang Chen, Taozhi Wang, Lecheng Tao, Yongding Gu, Bin Li, Ge Machine Learning Reinforcement learning with verifiable reward (RLVR) has become a promising paradigm for post-training large language models (LLMs) to improve their reasoning capability. However, when the rollout accuracy is low on hard problems, the reward becomes sparse, limiting learning efficiency and causing exploration bottlenecks. Existing approaches either rely on teacher models for distillation or filter out difficult problems, which limits scalability or restricts reasoning improvement through exploration. We propose EvoCoT, a self-evolving curriculum learning framework based on two-stage chain-of-thought (CoT) reasoning optimization. EvoCoT constrains the exploration space by self-generating and verifying CoT trajectories, then gradually shortens CoT steps to expand the space in a controlled way. The framework enables LLMs to stably learn from initially unsolved hard problems under sparse rewards. We apply EvoCoT to multiple LLM families, including Qwen, DeepSeek, and Llama. Experiments show that EvoCoT enables LLMs to solve previously unsolved problems, improves reasoning capability without external CoT supervision, and is compatible with various RL fine-tuning methods. We release the source code to support future research. |
| title | EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2508.07809 |