Cover Image

On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Agarwal, Rishabh, Vieillard, Nino, Zhou, Yongchao, Stanczyk, Piotr, Ramos, Sabela, Geist, Matthieu, Bachem, Olivier
Format:	Preprint
Published:	2023
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

BOND: Aligning LLMs with Best-of-N Distillation
by: Sessa, Pier Giuseppe, et al.
Published: (2024)

WARM: On the Benefits of Weight Averaged Reward Models
by: Ramé, Alexandre, et al.
Published: (2024)

On Teacher Hacking in Language Model Distillation
by: Tiapkin, Daniil, et al.
Published: (2025)

Imitating Language via Scalable Inverse Reinforcement Learning
by: Wulfmeier, Markus, et al.
Published: (2024)

Transformers Can Achieve Length Generalization But Not Robustly
by: Zhou, Yongchao, et al.
Published: (2024)

DistillSpec: Improving Speculative Decoding via Knowledge Distillation
by: Zhou, Yongchao, et al.
Published: (2023)

OPSDL: On-Policy Self-Distillation for Long-Context Language Models
by: Zhang, Xinsen, et al.
Published: (2026)

Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models
by: Noukhovitch, Michael, et al.
Published: (2024)

WARP: On the Benefits of Weight Averaged Rewarded Policies
by: Ramé, Alexandre, et al.
Published: (2024)

In-Context Principle Learning from Mistakes
by: Zhang, Tianjun, et al.
Published: (2024)

Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning
by: Wang, Kaiwen, et al.
Published: (2024)

BAGEL: Benchmarking Animal Knowledge Expertise in Language Models
by: Shen, Jiacheng, et al.
Published: (2026)

PLPP: Prompt Learning with Perplexity Is Self-Distillation for Vision-Language Models
by: Liu, Biao, et al.
Published: (2024)

MiniLLM: On-Policy Distillation of Large Language Models
by: Gu, Yuxian, et al.
Published: (2023)

Black-Box On-Policy Distillation of Large Language Models
by: Ye, Tianzhu, et al.
Published: (2025)

Learning From Mistakes Makes LLM Better Reasoner
by: An, Shengnan, et al.
Published: (2023)

The Impact of Steering Large Language Models with Persona Vectors in Educational Applications
by: Wu, Yongchao, et al.
Published: (2026)

Chunk-Distilled Language Modeling
by: Li, Yanhong, et al.
Published: (2024)

One-Step Flow Policy: Self-Distillation for Fast Visuomotor Policies
by: Li, Shaolong, et al.
Published: (2026)

Transparent and Coherent Procedural Mistake Detection
by: Storks, Shane, et al.
Published: (2024)

Towards Generating Informative Textual Description for Neurons in Language Models
by: Mondal, Shrayani, et al.
Published: (2024)

AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation
by: Zhang, Songming, et al.
Published: (2025)

Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems
by: Ye, Tian, et al.
Published: (2024)

Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis
by: Chen, Kai, et al.
Published: (2023)

Self-Knowledge Distillation for Learning Ambiguity
by: Park, Hancheol, et al.
Published: (2024)

Understanding Likelihood Over-optimisation in Direct Alignment Algorithms
by: Shi, Zhengyan, et al.
Published: (2024)

Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic
by: Bhardwaj, Rishabh, et al.
Published: (2024)

Copy-Paste to Mitigate Large Language Model Hallucinations
by: Long, Yongchao, et al.
Published: (2025)

VEPO: Variable Entropy Policy Optimization for Low-Resource Language Foundation Models
by: Liu, Chonghan, et al.
Published: (2026)

Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation
by: Yang, Wenkai, et al.
Published: (2026)

Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling
by: Xu, Wenda, et al.
Published: (2024)

DP-OPD: Differentially Private On-Policy Distillation for Language Models
by: Khadem, Fatemeh, et al.
Published: (2026)

UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models
by: Dong, Yijiang River, et al.
Published: (2024)

Mechanistic Indicators of Understanding in Large Language Models
by: Beckmann, Pierre, et al.
Published: (2025)

V-STaR: Training Verifiers for Self-Taught Reasoners
by: Hosseini, Arian, et al.
Published: (2024)

CEM: A Data-Efficient Method for Large Language Models to Continue Evolving From Mistakes
by: Zhao, Haokun, et al.
Published: (2024)

Same Evidence, Different Answers: Canonical-Context On-Policy Distillation for Multi-Turn Language Models
by: Lin, Zizhuo, et al.
Published: (2026)

Vocabulary Expansion of Large Language Models via Kullback-Leibler-Based Self-Distillation
by: Linder, Max Rehman
Published: (2025)

Beyond the Parameters: A Technical Survey of Contextual Enrichment in Large Language Models: From In-Context Prompting to Causal Retrieval-Augmented Generation
by: Bansal, Prakhar, et al.
Published: (2026)

Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
by: Li, Yaxuan, et al.
Published: (2026)