Are Full Rollouts Necessary for On-Policy Distillation?
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917554816024576 |
|---|---|
| author | Zhang, Yaocheng Chai, Jiajun Fu, Yuqian Tu, Songjun Wang, Xiaohan Lin, Wei Yin, Guojun Zhang, Qichao Zhu, Yuanheng Zhao, Dongbin |
| author_facet | Zhang, Yaocheng Chai, Jiajun Fu, Yuqian Tu, Songjun Wang, Xiaohan Lin, Wei Yin, Guojun Zhang, Qichao Zhu, Yuanheng Zhao, Dongbin |
| contents | On-policy distillation (OPD) provides dense teacher feedback along student-generated rollouts rather than fixed teacher traces and has emerged as a promising post-training paradigm. However, standard OPD typically generates full rollouts during training, which is computationally expensive and may expose the student to unreliable teacher feedback at late rollout positions, especially during early training. We identify the rollout horizon as a key bottleneck in OPD that substantially impacts training efficiency. Unlike Reinforcement Learning with Verifiable Rewards (RLVR), OPD does not require a final answer reward to provide learning signals. Therefore, full rollouts may not always be necessary for OPD. Motivated by this insight, we propose two simple horizon-control strategies: Progressive OPD (POPD), which gradually expands the rollout horizon during training, and Truncated OPD (TOPD), which permanently performs distillation on reliable truncated rollouts. Experiments on mathematical reasoning show that POPD improves the training efficiency of OPD by up to 3$\times$, while TOPD matches OPD performance using only 10\% of the rollout horizon, leading to substantial wall-clock and memory reductions. These results demonstrate that controlling the rollout horizon offers a simple and practical path to more efficient OPD. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_31490 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Are Full Rollouts Necessary for On-Policy Distillation? Zhang, Yaocheng Chai, Jiajun Fu, Yuqian Tu, Songjun Wang, Xiaohan Lin, Wei Yin, Guojun Zhang, Qichao Zhu, Yuanheng Zhao, Dongbin Computation and Language On-policy distillation (OPD) provides dense teacher feedback along student-generated rollouts rather than fixed teacher traces and has emerged as a promising post-training paradigm. However, standard OPD typically generates full rollouts during training, which is computationally expensive and may expose the student to unreliable teacher feedback at late rollout positions, especially during early training. We identify the rollout horizon as a key bottleneck in OPD that substantially impacts training efficiency. Unlike Reinforcement Learning with Verifiable Rewards (RLVR), OPD does not require a final answer reward to provide learning signals. Therefore, full rollouts may not always be necessary for OPD. Motivated by this insight, we propose two simple horizon-control strategies: Progressive OPD (POPD), which gradually expands the rollout horizon during training, and Truncated OPD (TOPD), which permanently performs distillation on reliable truncated rollouts. Experiments on mathematical reasoning show that POPD improves the training efficiency of OPD by up to 3$\times$, while TOPD matches OPD performance using only 10\% of the rollout horizon, leading to substantial wall-clock and memory reductions. These results demonstrate that controlling the rollout horizon offers a simple and practical path to more efficient OPD. |
| title | Are Full Rollouts Necessary for On-Policy Distillation? |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.31490 |