Cover Image

On-Policy Context Distillation for Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Ye, Tianzhu, Dong, Li, Wu, Xun, Huang, Shaohan, Wei, Furu
Format:	Preprint
Published:	2026
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Black-Box On-Policy Distillation of Large Language Models
by: Ye, Tianzhu, et al.
Published: (2025)

Online Experiential Learning for Language Models
by: Ye, Tianzhu, et al.
Published: (2026)

Textual Aesthetics in Large Language Models
by: Jiang, Lingjie, et al.
Published: (2024)

Universal YOCO for Efficient Depth Scaling
by: Sun, Yutao, et al.
Published: (2026)

Mixture of LoRA Experts
by: Wu, Xun, et al.
Published: (2024)

On-Policy RL with Optimal Reward Baseline
by: Hao, Yaru, et al.
Published: (2025)

BitNet Distillation
by: Wu, Xun, et al.
Published: (2025)

MiniLLM: On-Policy Distillation of Large Language Models
by: Gu, Yuxian, et al.
Published: (2023)

MH-MoE: Multi-Head Mixture-of-Experts
by: Huang, Shaohan, et al.
Published: (2024)

The Era of Agentic Organization: Learning to Organize with Language Models
by: Chi, Zewen, et al.
Published: (2025)

Adapting Large Language Models to Domains via Reading Comprehension
by: Cheng, Daixuan, et al.
Published: (2023)

Kosmos-G: Generating Images in Context with Multimodal Large Language Models
by: Pan, Xichen, et al.
Published: (2023)

Reward Reasoning Model
by: Guo, Jiaxin, et al.
Published: (2025)

Multi-Head Mixture-of-Experts
by: Wu, Xun, et al.
Published: (2024)

Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generation
by: Wu, Xun, et al.
Published: (2024)

VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models
by: Jiang, Lingjie, et al.
Published: (2025)

Instruction Pre-Training: Language Models are Supervised Multitask Learners
by: Cheng, Daixuan, et al.
Published: (2024)

Think Only When You Need with Large Hybrid-Reasoning Models
by: Jiang, Lingjie, et al.
Published: (2025)

Geometric-Mean Policy Optimization
by: Zhao, Yuzhong, et al.
Published: (2025)

Thinking Augmented Pre-training
by: Wang, Liang, et al.
Published: (2025)

Direct Preference Knowledge Distillation for Large Language Models
by: Li, Yixing, et al.
Published: (2024)

Reinforcement Pre-Training
by: Dong, Qingxiu, et al.
Published: (2025)

Code Aesthetics with Agentic Reward Feedback
by: Xiao, Bang, et al.
Published: (2025)

You Only Cache Once: Decoder-Decoder Architectures for Language Models
by: Sun, Yutao, et al.
Published: (2024)

Multimodal Latent Language Modeling with Next-Token Diffusion
by: Sun, Yutao, et al.
Published: (2024)

Differential Transformer
by: Ye, Tianzhu, et al.
Published: (2024)

Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts
by: Zhang, Di, et al.
Published: (2025)

$Se^2$: Sequential Example Selection for In-Context Learning
by: Liu, Haoyu, et al.
Published: (2024)

Context-DPO: Aligning Language Models for Context-Faithfulness
by: Bi, Baolong, et al.
Published: (2024)

Learning to Retrieve In-Context Examples for Large Language Models
by: Wang, Liang, et al.
Published: (2023)

Towards Optimal Learning of Language Models
by: Gu, Yuxian, et al.
Published: (2024)

HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition
by: Liu, Yuxuan, et al.
Published: (2024)

In-context Autoencoder for Context Compression in a Large Language Model
by: Ge, Tao, et al.
Published: (2023)

WildLong: Synthesizing Realistic Long-Context Instruction Data at Scale
by: Li, Jiaxi, et al.
Published: (2025)

The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
by: Ma, Shuming, et al.
Published: (2024)

Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
by: Li, Haoran, et al.
Published: (2024)

Breaking Training Bottlenecks: Effective and Stable Reinforcement Learning for Coding Models
by: Li, Zongqian, et al.
Published: (2026)

Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity
by: Zhang, Di, et al.
Published: (2026)

Data Selection via Optimal Control for Language Models
by: Gu, Yuxian, et al.
Published: (2024)

Rectified Sparse Attention
by: Sun, Yutao, et al.
Published: (2025)