Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917547218042880 |
|---|---|
| author | Liu, Yanjiang Lou, Jie Guan, Xinyan Ji, Yuqiu Lin, Hongyu He, Ben Han, Xianpei Sun, Le Yu, Xing Lu, Yaojie |
| author_facet | Liu, Yanjiang Lou, Jie Guan, Xinyan Ji, Yuqiu Lin, Hongyu He, Ben Han, Xianpei Sun, Le Yu, Xing Lu, Yaojie |
| contents | On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher. However, we identify a critical bottleneck, \textbf{Supervision Fidelity Decay (SFD)}: as student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative. Consequently, the teacher-dependent corrective signal in reverse-KL distillation weakens, causing student drift to compound across long reasoning chains. To mitigate SFD, we introduce \textbf{Lookahead Group Reward (\ours{})}. Building on the insight that next-step teacher confidence reflects the discriminative strength of future reverse-KL supervision, \ours{} evaluates the student's top-K candidate tokens by the teacher confidence they induce at the subsequent step and assigns a group-normalized reward. To maintain computational efficiency, we further design an entropy-triggered tree-attention mechanism. Across six math and code benchmarks, \ours{} improves mean@8 by \textbf{2.57} points over OPD for a 7B student, with gains increasing in longer-generation and reaching +\textbf{4.92} points on AIME-26 at 39k tokens. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_30833 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation Liu, Yanjiang Lou, Jie Guan, Xinyan Ji, Yuqiu Lin, Hongyu He, Ben Han, Xianpei Sun, Le Yu, Xing Lu, Yaojie Computation and Language Artificial Intelligence On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher. However, we identify a critical bottleneck, \textbf{Supervision Fidelity Decay (SFD)}: as student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative. Consequently, the teacher-dependent corrective signal in reverse-KL distillation weakens, causing student drift to compound across long reasoning chains. To mitigate SFD, we introduce \textbf{Lookahead Group Reward (\ours{})}. Building on the insight that next-step teacher confidence reflects the discriminative strength of future reverse-KL supervision, \ours{} evaluates the student's top-K candidate tokens by the teacher confidence they induce at the subsequent step and assigns a group-normalized reward. To maintain computational efficiency, we further design an entropy-triggered tree-attention mechanism. Across six math and code benchmarks, \ours{} improves mean@8 by \textbf{2.57} points over OPD for a 7B student, with gains increasing in longer-generation and reaching +\textbf{4.92} points on AIME-26 at 39k tokens. |
| title | Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2605.30833 |