Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2605.12741 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911677331537920 |
|---|---|
| author | Zhang, Yuwei Li, Sha Yu, Changlong Lu, Qin Jin, Shuowei Dong, Chengyu Liu, Haoran Hong, Ilgee Li, Xintong Shi, Zhenyu Yin, Bing Shang, Jingbo |
| author_facet | Zhang, Yuwei Li, Sha Yu, Changlong Lu, Qin Jin, Shuowei Dong, Chengyu Liu, Haoran Hong, Ilgee Li, Xintong Shi, Zhenyu Yin, Bing Shang, Jingbo |
| contents | Enabling Large Language Models (LLMs) to continuously improve from environmental interactions is a central challenge in post-training. While on-policy self-distillation offers a promising paradigm, existing methods predominantly treat environmental feedback as a passive conditioning signal. Consequently, they heavily rely on successful demonstrations and struggle to learn in rare-success regimes. To bridge this gap, we introduce Reflection-Enhanced Self-Distillation (RESD), a framework that transforms raw failure feedback into an active source of corrective supervision. Instead of passively appending feedback, RESD interprets failed trajectories by generating retrospective reflections to diagnose local errors, and curates a persistent global playbook to preserve reusable lessons across training steps. The enriched context enables the self-teacher to provide actionable token-level supervision even in the absence of successful rollouts. Empirical evaluations on multiple continual learning tasks demonstrate that RESD substantially outperforms standard self-distillation baselines. Furthermore, RESD achieves significantly faster early-stage improvement than GRPO with $8\times$ samples using only a single rollout per prompt, highlighting its superior interaction efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_12741 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillation Zhang, Yuwei Li, Sha Yu, Changlong Lu, Qin Jin, Shuowei Dong, Chengyu Liu, Haoran Hong, Ilgee Li, Xintong Shi, Zhenyu Yin, Bing Shang, Jingbo Machine Learning Enabling Large Language Models (LLMs) to continuously improve from environmental interactions is a central challenge in post-training. While on-policy self-distillation offers a promising paradigm, existing methods predominantly treat environmental feedback as a passive conditioning signal. Consequently, they heavily rely on successful demonstrations and struggle to learn in rare-success regimes. To bridge this gap, we introduce Reflection-Enhanced Self-Distillation (RESD), a framework that transforms raw failure feedback into an active source of corrective supervision. Instead of passively appending feedback, RESD interprets failed trajectories by generating retrospective reflections to diagnose local errors, and curates a persistent global playbook to preserve reusable lessons across training steps. The enriched context enables the self-teacher to provide actionable token-level supervision even in the absence of successful rollouts. Empirical evaluations on multiple continual learning tasks demonstrate that RESD substantially outperforms standard self-distillation baselines. Furthermore, RESD achieves significantly faster early-stage improvement than GRPO with $8\times$ samples using only a single rollout per prompt, highlighting its superior interaction efficiency. |
| title | Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillation |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2605.12741 |