LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908572942598144 |
|---|---|
| author | Chen, Weizhe Koenig, Sven Dilkina, Bistra |
| author_facet | Chen, Weizhe Koenig, Sven Dilkina, Bistra |
| contents | Since the release of Deepseek-R1, reinforcement learning with verifiable rewards (RLVR) has become a central approach for training large language models (LLMs) on reasoning tasks. Recent work has largely focused on modifying loss functions to make RLVR more efficient and effective. In this paper, motivated by studies of overthinking in LLMs, we propose Length-aware Sampling for Policy Optimization (LSPO), a novel meta-RLVR algorithm that dynamically selects training data at each step based on the average response length. We evaluate LSPO across multiple base models and datasets, demonstrating that it consistently improves learning effectiveness. In addition, we conduct a detailed ablation study to examine alternative ways of incorporating length signals into dynamic sampling, offering further insights and highlighting promising directions for future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_01459 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoning Chen, Weizhe Koenig, Sven Dilkina, Bistra Machine Learning Computation and Language Since the release of Deepseek-R1, reinforcement learning with verifiable rewards (RLVR) has become a central approach for training large language models (LLMs) on reasoning tasks. Recent work has largely focused on modifying loss functions to make RLVR more efficient and effective. In this paper, motivated by studies of overthinking in LLMs, we propose Length-aware Sampling for Policy Optimization (LSPO), a novel meta-RLVR algorithm that dynamically selects training data at each step based on the average response length. We evaluate LSPO across multiple base models and datasets, demonstrating that it consistently improves learning effectiveness. In addition, we conduct a detailed ablation study to examine alternative ways of incorporating length signals into dynamic sampling, offering further insights and highlighting promising directions for future research. |
| title | LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoning |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2510.01459 |