Cover Image

Efficient Knowledge Injection in LLMs via Self-Distillation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Kujanpää, Kalle, Marttinen, Pekka, Valpola, Harri, Ilin, Alexander
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Edit Knowledge, Not Just Facts via Multi-Step Reasoning over Background Stories
by: Gao, Ya, et al.
Published: (2026)

Memento No More: Coaching AI Agents to Master Multiple Tasks via Hints Internalization
by: Alakuijala, Minttu, et al.
Published: (2025)

Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities
by: Nikitin, Alexander, et al.
Published: (2024)

Improved Compositional Generalization by Generating Demonstrations for Meta-Learning
by: Spilsbury, Sam, et al.
Published: (2023)

Object-level Self-Distillation for Vision Pretraining
by: Hızlı, Çağlar, et al.
Published: (2025)

In-Context Symbolic Regression: Leveraging Large Language Models for Function Discovery
by: Merler, Matteo, et al.
Published: (2024)

Query-Guided Self-Supervised Summarization of Nursing Notes
by: Gao, Ya, et al.
Published: (2024)

Diffusion models as probabilistic neural operators for recovering unobserved states of dynamical systems
by: Haitsiukevich, Katsiaryna, et al.
Published: (2024)

Beyond Autoregression: Fast LLMs via Self-Distillation Through Time
by: Deschenaux, Justin, et al.
Published: (2024)

Improving Medical Multi-modal Contrastive Learning with Expert Annotations
by: Kumar, Yogesh, et al.
Published: (2024)

Breaking the Attention Bottleneck
by: Hilsenbek, Kalle
Published: (2024)

Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs
by: Ovadia, Oded, et al.
Published: (2023)

Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
by: Kim, Jeonghye, et al.
Published: (2026)

Distilled Self-Critique of LLMs with Synthetic Data: a Bayesian Perspective
by: Gallego, Victor
Published: (2023)

Language Model Knowledge Distillation for Efficient Question Answering in Spanish
by: Bazaga, Adrián, et al.
Published: (2023)

Knowledge-augmented Graph Neural Networks with Concept-aware Attention for Adverse Drug Event Detection
by: Ji, Shaoxiong, et al.
Published: (2023)

Efficiently Distilling LLMs for Edge Applications
by: Kundu, Achintya, et al.
Published: (2024)

Learning to Staff: Offline Reinforcement Learning and Fine-Tuned LLMs for Warehouse Staffing Optimization
by: Kujanpää, Kalle, et al.
Published: (2026)

Efficient Mathematical Reasoning Models via Dynamic Pruning and Knowledge Distillation
by: Yu, Fengming, et al.
Published: (2025)

Multi-Token Prediction via Self-Distillation
by: Kirchenbauer, John, et al.
Published: (2026)

Exploring the Limits of Model Compression in LLMs: A Knowledge Distillation Study on QA Tasks
by: Datta, Joyeeta, et al.
Published: (2025)

DRAG: Distilling RAG for SLMs from LLMs to Transfer Knowledge and Mitigate Hallucination via Evidence and Graph-based Distillation
by: Chen, Jennifer, et al.
Published: (2025)

KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning
by: Xu, Hongling, et al.
Published: (2025)

Continuous Monte Carlo Graph Search
by: Kujanpää, Kalle, et al.
Published: (2022)

Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge with Curriculum Preference Learning
by: Wang, Xiyao, et al.
Published: (2024)

Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMs
by: Cha, Sungmin, et al.
Published: (2024)

Few-Shot Knowledge Distillation of LLMs With Counterfactual Explanations
by: Hamman, Faisal, et al.
Published: (2025)

Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
by: Zhao, Siyan, et al.
Published: (2026)

DistillSpec: Improving Speculative Decoding via Knowledge Distillation
by: Zhou, Yongchao, et al.
Published: (2023)

Knowledge Distillation with Training Wheels
by: Liu, Guanlin, et al.
Published: (2025)

Self-Distilled RLVR
by: Yang, Chenxu, et al.
Published: (2026)

Efficient Data Selection at Scale via Influence Distillation
by: Nikdan, Mahdi, et al.
Published: (2025)

Scaling Reasoning Efficiently via Relaxed On-Policy Distillation
by: Ko, Jongwoo, et al.
Published: (2026)

Self-Refining Language Model Anonymizers via Adversarial Distillation
by: Kim, Kyuyoung, et al.
Published: (2025)

Confidence-aware Self-Semantic Distillation on Knowledge Graph Embedding
by: Liu, Yichen, et al.
Published: (2022)

MeKi: Memory-based Expert Knowledge Injection for Efficient LLM Scaling
by: Ding, Ning, et al.
Published: (2026)

Sinkhorn Distance Minimization for Knowledge Distillation
by: Cui, Xiao, et al.
Published: (2024)

From Style to Facts: Mapping the Boundaries of Knowledge Injection with Finetuning
by: Zhao, Eric, et al.
Published: (2025)

Self-Calibrating Language Models via Test-Time Discriminative Distillation
by: Hedna, Mohamed Rissal, et al.
Published: (2026)

Few-Step Diffusion Language Models via Trajectory Self-Distillation
by: Zhang, Tunyu, et al.
Published: (2026)