Image de couverture de livre

Warmup-Distill: Bridge the Distribution Mismatch between Teacher and Student before Knowledge Distillation

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Sun, Zengkui, Liu, Yijin, Meng, Fandong, Chen, Yufeng, Xu, Jinan, Zhou, Jie
Format:	Preprint
Publié:	2025
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Enhancing Cross-Tokenizer Knowledge Distillation with Contextual Dynamical Mapping
par: Chen, Yijie, et autres
Publié: (2025)

LCS: A Language Converter Strategy for Zero-Shot Neural Machine Translation
par: Sun, Zengkui, et autres
Publié: (2024)

Outdated Issue Aware Decoding for Reasoning Questions on Edited Knowledge
par: Sun, Zengkui, et autres
Publié: (2024)

Dual-Space Knowledge Distillation for Large Language Models
par: Zhang, Songming, et autres
Publié: (2024)

A Dual-Space Framework for General Knowledge Distillation of Large Language Models
par: Zhang, Xue, et autres
Publié: (2025)

Beyond Binary Gender: Evaluating Gender-Inclusive Machine Translation with Ambiguous Attitude Words
par: Chen, Yijie, et autres
Publié: (2024)

Comments as Natural Logic Pivots: Improve Code Generation via Comment Perspective
par: Chen, Yijie, et autres
Publié: (2024)

Multilingual Knowledge Editing with Language-Agnostic Factual Neurons
par: Zhang, Xue, et autres
Publié: (2024)

MiniPLM: Knowledge Distillation for Pre-Training Language Models
par: Gu, Yuxian, et autres
Publié: (2024)

Accelerating Inference in Large Language Models with a Unified Layer Skipping Strategy
par: Liu, Yijin, et autres
Publié: (2024)

Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation
par: Zhang, Songming, et autres
Publié: (2023)

Cross-Lingual Knowledge Editing in Large Language Models
par: Wang, Jiaan, et autres
Publié: (2023)

Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts
par: Zhang, Xue, et autres
Publié: (2025)

CM-Align: Consistency-based Multilingual Alignment for Large Language Models
par: Zhang, Xue, et autres
Publié: (2025)

Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling
par: Xu, Wenda, et autres
Publié: (2024)

SED-SFT: Selectively Encouraging Diversity in Supervised Fine-Tuning
par: Chen, Yijie, et autres
Publié: (2026)

AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation
par: Zhang, Songming, et autres
Publié: (2025)

Language Generation with Strictly Proper Scoring Rules
par: Shao, Chenze, et autres
Publié: (2024)

KDFlow: A User-Friendly and Efficient Knowledge Distillation Framework for Large Language Models
par: Zhang, Songming, et autres
Publié: (2026)

Multi-Aspect Knowledge Distillation for Language Model with Low-rank Factorization
par: Liu, Zihe, et autres
Publié: (2026)

Think Natively: Unlocking Multilingual Reasoning with Consistency-Enhanced Reinforcement Learning
par: Zhang, Xue, et autres
Publié: (2025)

An Empirical Study of Many-to-Many Summarization with Large Language Models
par: Wang, Jiaan, et autres
Publié: (2025)

Figure It Out: Improve the Frontier of Reasoning with Executable Visual States
par: Chen, Meiqi, et autres
Publié: (2025)

Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMs
par: Jin, Ruihan, et autres
Publié: (2026)

Retrieval-Augmented Machine Translation with Unstructured Knowledge
par: Wang, Jiaan, et autres
Publié: (2024)

In Good GRACEs: Principled Teacher Selection for Knowledge Distillation
par: Panigrahi, Abhishek, et autres
Publié: (2025)

Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning
par: Yang, Zhaorui, et autres
Publié: (2024)

Dual-Space Knowledge Distillation with Key-Query Matching for Large Language Models with Vocabulary Mismatch
par: Tsiapali, Stella Eva, et autres
Publié: (2026)

Progressive Residual Warmup for Language Model Pretraining
par: Chen, Tianhao, et autres
Publié: (2026)

THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translation
par: Liang, Yunlong, et autres
Publié: (2025)

Distilling Rule-based Knowledge into Large Language Models
par: Yang, Wenkai, et autres
Publié: (2023)

D2C: Unlocking the Potential of Continuous Autoregressive Image Generation with Discrete Tokens
par: Wang, Panpan, et autres
Publié: (2025)

SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Models
par: Koo, Jahyun, et autres
Publié: (2024)

ReasoningRank: Teaching Student Models to Rank through Reasoning-Based Knowledge Distillation
par: Ji, Yuelyu, et autres
Publié: (2024)

TwT: Thinking without Tokens by Habitual Reasoning Distillation with Multi-Teachers' Guidance
par: Xu, Jingxian, et autres
Publié: (2025)

Sinkhorn Distance Minimization for Knowledge Distillation
par: Cui, Xiao, et autres
Publié: (2024)

When Better Teachers Don't Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQA
par: Tuchinda, Pume, et autres
Publié: (2025)

DeepTrans: Deep Reasoning Translation via Reinforcement Learning
par: Wang, Jiaan, et autres
Publié: (2025)

ExTrans: Multilingual Deep Reasoning Translation via Exemplar-Enhanced Reinforcement Learning
par: Wang, Jiaan, et autres
Publié: (2025)

CSCD-NS: a Chinese Spelling Check Dataset for Native Speakers
par: Hu, Yong, et autres
Publié: (2022)