Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911578814676992 |
|---|---|
| author | Luo, Feng Chuang, Yu-Neng Wang, Guanchu Xu, Zicheng Han, Xiaotian Zhang, Tianyi Braverman, Vladimir |
| author_facet | Luo, Feng Chuang, Yu-Neng Wang, Guanchu Xu, Zicheng Han, Xiaotian Zhang, Tianyi Braverman, Vladimir |
| contents | On-policy distillation (OPD) trains student models under their own induced distribution while leveraging supervision from stronger teachers. We identify a failure mode of OPD: as training progresses, on-policy rollouts can undergo abrupt length inflation, causing truncated trajectories to dominate the training data. This truncation collapse coincides with abrupt repetition saturation and induces biased gradient signals, leading to severe training instability and sharp degradation in validation performance. We attribute this problem to the interaction between student-induced data collection and the distillation objective, which implicitly favors long and repetitive rollouts. To address this issue, we propose StableOPD, a stabilized OPD framework that combines a reference-based divergence constraint with rollout mixture distillation. These together mitigate repetition-induced length inflation and further stabilize OPD training. Across multiple math reasoning datasets, our approach prevents truncation collapse, stabilizes training dynamics, and improves performance by 7.2% on average. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_08527 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models Luo, Feng Chuang, Yu-Neng Wang, Guanchu Xu, Zicheng Han, Xiaotian Zhang, Tianyi Braverman, Vladimir Computation and Language Machine Learning On-policy distillation (OPD) trains student models under their own induced distribution while leveraging supervision from stronger teachers. We identify a failure mode of OPD: as training progresses, on-policy rollouts can undergo abrupt length inflation, causing truncated trajectories to dominate the training data. This truncation collapse coincides with abrupt repetition saturation and induces biased gradient signals, leading to severe training instability and sharp degradation in validation performance. We attribute this problem to the interaction between student-induced data collection and the distillation objective, which implicitly favors long and repetitive rollouts. To address this issue, we propose StableOPD, a stabilized OPD framework that combines a reference-based divergence constraint with rollout mixture distillation. These together mitigate repetition-induced length inflation and further stabilize OPD training. Across multiple math reasoning datasets, our approach prevents truncation collapse, stabilizes training dynamics, and improves performance by 7.2% on average. |
| title | Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2604.08527 |