Cover Image

Honey, I Shrunk the Language Model: Impact of Knowledge Distillation Methods on Performance and Explainability

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Hendriks, Daniel, Spitzer, Philipp, Kühl, Niklas, Satzger, Gerhard
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Towards Human-Understandable Multi-Dimensional Concept Discovery
by: Grobrügge, Arne, et al.
Published: (2025)

Transferring Domain Knowledge with (X)AI-Based Learning Systems
by: Spitzer, Philipp, et al.
Published: (2024)

Honey, I Shrunk the Arc de Triomphe!
by: Xiangli, Yuanbo, et al.
Published: (2026)

Don't be Fooled: The Misinformation Effect of Explanations in Human-AI Collaboration
by: Spitzer, Philipp, et al.
Published: (2024)

Human Delegation Behavior in Human-AI Collaboration: The Effect of Contextual Information
by: Spitzer, Philipp, et al.
Published: (2024)

From Model Uncertainty to Human Attention: Localization-Aware Visual Cues for Scalable Annotation Review
by: Sbeyti, Moussa Kassem, et al.
Published: (2026)

Data Quality Challenges in Retrieval-Augmented Generation
by: Müller, Leopold, et al.
Published: (2025)

Complementarity in Human-AI Collaboration: Concept, Sources, and Evidence
by: Hemmer, Patrick, et al.
Published: (2024)

Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application
by: Yang, Chuanpeng, et al.
Published: (2024)

The Impact of Imperfect XAI on Human-AI Decision-Making
by: Morrison, Katelyn, et al.
Published: (2023)

Memorization Dynamics in Knowledge Distillation for Language Models
by: Borkar, Jaydeep, et al.
Published: (2026)

Revisiting Knowledge Distillation for Autoregressive Language Models
by: Zhong, Qihuang, et al.
Published: (2024)

Improved Methods for Model Pruning and Knowledge Distillation
by: Jiang, Wei, et al.
Published: (2025)

Knowledge Distillation for Temporal Knowledge Graph Reasoning with Large Language Models
by: Xing, Wang, et al.
Published: (2026)

Multi-Sense Embeddings for Language Models and Knowledge Distillation
by: Wang, Qitong, et al.
Published: (2025)

Knowledge Distillation of Black-Box Large Language Models
by: Chen, Hongzhan, et al.
Published: (2024)

Distilling Rule-based Knowledge into Large Language Models
by: Yang, Wenkai, et al.
Published: (2023)

Direct Preference Knowledge Distillation for Large Language Models
by: Li, Yixing, et al.
Published: (2024)

A Survey on Knowledge Distillation of Large Language Models
by: Xu, Xiaohan, et al.
Published: (2024)

On the Performance of an Explainable Language Model on PubMedQA
by: Srinivasan, Venkat, et al.
Published: (2025)

Understanding Data Understanding: A Framework to Navigate the Intricacies of Data Analytics
by: Holstein, Joshua, et al.
Published: (2024)

SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Models
by: Koo, Jahyun, et al.
Published: (2024)

Evolving Knowledge Distillation with Large Language Models and Active Learning
by: Liu, Chengyuan, et al.
Published: (2024)

MiniPLM: Knowledge Distillation for Pre-Training Language Models
by: Gu, Yuxian, et al.
Published: (2024)

DDK: Distilling Domain Knowledge for Efficient Large Language Models
by: Liu, Jiaheng, et al.
Published: (2024)

Native Design Bias: Studying the Impact of English Nativeness on Language Model Performance
by: Reusens, Manon, et al.
Published: (2024)

Leveraging Large Language Models for Enhanced NLP Task Performance through Knowledge Distillation and Optimized Training Strategies
by: Huang, Yining, et al.
Published: (2024)

BabyLMs for isiXhosa: Data-Efficient Language Modelling in a Low-Resource Context
by: Matzopoulos, Alexis, et al.
Published: (2025)

Context versus Prior Knowledge in Language Models
by: Du, Kevin, et al.
Published: (2024)

Efficient Knowledge Distillation: Empowering Small Language Models with Teacher Model Insights
by: Ballout, Mohamad, et al.
Published: (2024)

The Valley of Code Reasoning: Scaling Knowledge Distillation of Large Language Models
by: He, Muyu, et al.
Published: (2025)

Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models
by: Rao, Jun, et al.
Published: (2024)

Multi-Aspect Knowledge Distillation for Language Model with Low-rank Factorization
by: Liu, Zihe, et al.
Published: (2026)

Knowledge-Augmented Large Language Model Agents for Explainable Financial Decision-Making
by: Zhang, Qingyuan, et al.
Published: (2025)

Dual-Space Knowledge Distillation for Large Language Models
by: Zhang, Songming, et al.
Published: (2024)

ConStat: Performance-Based Contamination Detection in Large Language Models
by: Dekoninck, Jasper, et al.
Published: (2024)

Knowledge Distillation for Large Language Models
by: La Torre, Alejandro Paredes, et al.
Published: (2026)

Evaluating Explainable AI Attribution Methods in Neural Machine Translation via Attention-Guided Knowledge Distillation
by: Nourbakhsh, Aria, et al.
Published: (2026)

Development of Mental Models in Human-AI Collaboration: A Conceptual Framework
by: Holstein, Joshua, et al.
Published: (2025)

EasyDistill: A Comprehensive Toolkit for Effective Knowledge Distillation of Large Language Models
by: Wang, Chengyu, et al.
Published: (2025)