Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910215304118272 |
|---|---|
| author | Jia, Nan Yang, Haojin Ma, Xing Lian, Jiesong Zhang, Shuailiang Zhang, Weipeng Zeng, Ke Cai, Xunliang Sun, Zequn |
| author_facet | Jia, Nan Yang, Haojin Ma, Xing Lian, Jiesong Zhang, Shuailiang Zhang, Weipeng Zeng, Ke Cai, Xunliang Sun, Zequn |
| contents | On-policy distillation (OPD) trains a student on its own trajectories with token-level teacher feedback and often outperforms off-policy distillation and standard reinforcement learning. However, we find that its standard advantage weighted policy gradient suffers from three structural weaknesses, including high variance updates, vanishing gradients in zero-advantage regions, and exploration bottlenecks when corrective signals are insufficient. We therefore propose Asymmetric On-Policy Distillation (AOPD), which replaces ineffective negative reinforcement with localized divergence minimization in non-positive advantage regions while preserving positive reinforcement learning. Experiments on mathematical reasoning benchmarks show that AOPD consistently outperforms standard OPD, with average gains of 4.09 / 8.34 under strong / weak initialization, respectively. AOPD also maintains higher policy entropy during training and better capability retention during sequential tool-use adaptation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_06387 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level Jia, Nan Yang, Haojin Ma, Xing Lian, Jiesong Zhang, Shuailiang Zhang, Weipeng Zeng, Ke Cai, Xunliang Sun, Zequn Machine Learning Artificial Intelligence On-policy distillation (OPD) trains a student on its own trajectories with token-level teacher feedback and often outperforms off-policy distillation and standard reinforcement learning. However, we find that its standard advantage weighted policy gradient suffers from three structural weaknesses, including high variance updates, vanishing gradients in zero-advantage regions, and exploration bottlenecks when corrective signals are insufficient. We therefore propose Asymmetric On-Policy Distillation (AOPD), which replaces ineffective negative reinforcement with localized divergence minimization in non-positive advantage regions while preserving positive reinforcement learning. Experiments on mathematical reasoning benchmarks show that AOPD consistently outperforms standard OPD, with average gains of 4.09 / 8.34 under strong / weak initialization, respectively. AOPD also maintains higher policy entropy during training and better capability retention during sequential tool-use adaptation. |
| title | Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2605.06387 |