Cover Image

Knowledge Distillation with Training Wheels

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Liu, Guanlin, Ramachandran, Anand, Gangwani, Tanmay, Fu, Yan, Sethy, Abhinav
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Zero-Training Temporal Drift Detection for Transformer Sentiment Models: A Comprehensive Analysis on Authentic Social Media Streams
by: Bansal, Aayam, et al.
Published: (2025)

Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels
by: Pangakis, Nicholas, et al.
Published: (2024)

Brewing Knowledge in Context: Distillation Perspectives on In-Context Learning
by: Li, Chengye, et al.
Published: (2025)

No Training Wheels: Steering Vectors for Bias Correction at Inference Time
by: Gupta, Aviral, et al.
Published: (2025)

Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
by: Fang, Luyang, et al.
Published: (2025)

In Good GRACEs: Principled Teacher Selection for Knowledge Distillation
by: Panigrahi, Abhishek, et al.
Published: (2025)

Sinkhorn Distance Minimization for Knowledge Distillation
by: Cui, Xiao, et al.
Published: (2024)

Confidence-aware Self-Semantic Distillation on Knowledge Graph Embedding
by: Liu, Yichen, et al.
Published: (2022)

Few-Shot Knowledge Distillation of LLMs With Counterfactual Explanations
by: Hamman, Faisal, et al.
Published: (2025)

LoCa: Logit Calibration for Knowledge Distillation
by: Yang, Runming, et al.
Published: (2024)

Confidence Preservation Property in Knowledge Distillation Abstractions
by: Vengertsev, Dmitry, et al.
Published: (2024)

KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning
by: Xu, Hongling, et al.
Published: (2025)

Efficient Knowledge Injection in LLMs via Self-Distillation
by: Kujanpää, Kalle, et al.
Published: (2024)

Using Domain Knowledge to Guide Dialog Structure Induction via Neural Probabilistic Soft Logic
by: Pryor, Connor, et al.
Published: (2024)

Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models
by: Yang, Junjie, et al.
Published: (2025)

Winning Big with Small Models: Knowledge Distillation vs. Self-Training for Reducing Hallucination in Product QA Agents
by: Lewis, Ashley, et al.
Published: (2025)

Relational Knowledge Distillation Using Fine-tuned Function Vectors
by: Kang, Andrea, et al.
Published: (2026)

Tailoring Instructions to Student's Learning Levels Boosts Knowledge Distillation
by: Ren, Yuxin, et al.
Published: (2023)

A Survey on Symbolic Knowledge Distillation of Large Language Models
by: Acharya, Kamal, et al.
Published: (2024)

Reinforcement Learning-based Knowledge Distillation with LLM-as-a-Judge
by: Shen, Yiyang, et al.
Published: (2026)

Language Model Knowledge Distillation for Efficient Question Answering in Spanish
by: Bazaga, Adrián, et al.
Published: (2023)

Short Data, Long Context: Distilling Positional Knowledge in Transformers
by: Huber, Patrick, et al.
Published: (2026)

X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation
by: Sreenivas, Sharath Turuvekere, et al.
Published: (2026)

DistillSpec: Improving Speculative Decoding via Knowledge Distillation
by: Zhou, Yongchao, et al.
Published: (2023)

Knowledge Distillation from Large Language Models for Household Energy Modeling
by: Takrouri, Mohannad, et al.
Published: (2025)

Smoothing Out Hallucinations: Mitigating LLM Hallucination with Smoothed Knowledge Distillation
by: Nguyen, Hieu, et al.
Published: (2025)

Implicit Word Reordering with Knowledge Distillation for Cross-Lingual Dependency Parsing
by: Li, Zhuoran, et al.
Published: (2025)

HateTinyLLM : Hate Speech Detection Using Tiny Large Language Models
by: Sen, Tanmay, et al.
Published: (2024)

Multi-modal Anchor Gated Transformer with Knowledge Distillation for Emotion Recognition in Conversation
by: Li, Jie, et al.
Published: (2025)

HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Models
by: Lee, Seanie, et al.
Published: (2024)

MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models
by: Zhang, Ying, et al.
Published: (2024)

Attention to Mamba: A Recipe for Cross-Architecture Distillation
by: Moudgil, Abhinav, et al.
Published: (2026)

BitNet Distillation
by: Wu, Xun, et al.
Published: (2025)

Exploring the Limits of Model Compression in LLMs: A Knowledge Distillation Study on QA Tasks
by: Datta, Joyeeta, et al.
Published: (2025)

Delta Knowledge Distillation for Large Language Models
by: Cao, Yihan, et al.
Published: (2025)

DOGe: Defensive Output Generation for LLM Protection Against Knowledge Distillation
by: Li, Pingzhi, et al.
Published: (2025)

Leveraging Knowledge Distillation for Efficient Deep Reinforcement Learning in Resource-Constrained Environments
by: Meng, Guanlin
Published: (2023)

Can We Use Probing to Better Understand Fine-tuning and Knowledge Distillation of the BERT NLU?
by: Hościłowicz, Jakub, et al.
Published: (2023)

Gumbel Distillation for Parallel Text Generation
by: Zhang, Chi, et al.
Published: (2026)

Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
by: Zhao, Siyan, et al.
Published: (2026)