Cover Image

Iterative Layer-wise Distillation for Efficient Compression of Large Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Kovalev, Grigory, Tikhomirov, Mikhail
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

CharED: Character-wise Ensemble Decoding for Large Language Models
by: Gu, Kevin, et al.
Published: (2024)

Building Russian Benchmark for Evaluation of Information Retrieval Models
by: Kovalev, Grigory, et al.
Published: (2025)

Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression
by: Wang, Jingcun, et al.
Published: (2024)

GRASS: Gradient-based Adaptive Layer-wise Importance Sampling for Memory-efficient Large Language Model Fine-tuning
by: Tian, Kaiyuan, et al.
Published: (2026)

Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Models
by: Yao, Kai, et al.
Published: (2024)

MiniDisc: Minimal Distillation Schedule for Language Model Compression
by: Zhang, Chen, et al.
Published: (2022)

Wikipedia-based Datasets in Russian Information Retrieval Benchmark RusBEIR
by: Kovalev, Grigory, et al.
Published: (2025)

Thanos: A Block-wise Pruning Algorithm for Efficient Large Language Model Compression
by: Ilin, Ivan, et al.
Published: (2025)

Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initialization
by: He, Junlin, et al.
Published: (2026)

Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
by: Zhao, Siyan, et al.
Published: (2026)

On the Effect of Uncertainty on Layer-wise Inference Dynamics
by: Kim, Sunwoo, et al.
Published: (2025)

ILRe: Intermediate Layer Retrieval for Context Compression in Causal Language Models
by: Liang, Manlai, et al.
Published: (2025)

GRASP: Replace Redundant Layers with Adaptive Singular Parameters for Efficient Model Compression
by: Liu, Kainan, et al.
Published: (2024)

PIP: Perturbation-based Iterative Pruning for Large Language Models
by: Cao, Yi, et al.
Published: (2025)

From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Prediction
by: Zhu, Mingcheng, et al.
Published: (2026)

A Survey of On-Policy Distillation for Large Language Models
by: Song, Mingyang, et al.
Published: (2026)

Adversarial Moment-Matching Distillation of Large Language Models
by: Jia, Chen
Published: (2024)

SDMPrune: Self-Distillation MLP Pruning for Efficient Large Language Models
by: Zhu, Hourun, et al.
Published: (2025)

LLM-NEO: Parameter Efficient Knowledge Distillation for Large Language Models
by: Yang, Runming, et al.
Published: (2024)

LEAP: Layer-wise Exit-Aware Pretraining for Efficient Transformer Inference
by: Kapadia, Shashank, et al.
Published: (2026)

Investigating Layer Importance in Large Language Models
by: Zhang, Yang, et al.
Published: (2024)

Language Model Knowledge Distillation for Efficient Question Answering in Spanish
by: Bazaga, Adrián, et al.
Published: (2023)

Knowledge Distillation from Large Language Models for Household Energy Modeling
by: Takrouri, Mohannad, et al.
Published: (2025)

LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
by: Pan, Zhuoshi, et al.
Published: (2024)

Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
by: Fang, Luyang, et al.
Published: (2025)

SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillation
by: Li, Zichong, et al.
Published: (2025)

LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Models
by: Li, Guangyan, et al.
Published: (2024)

Cache & Distil: Optimising API Calls to Large Language Models
by: Ramírez, Guillem, et al.
Published: (2023)

A Survey on Symbolic Knowledge Distillation of Large Language Models
by: Acharya, Kamal, et al.
Published: (2024)

Distilling Large Language Models for Text-Attributed Graph Learning
by: Pan, Bo, et al.
Published: (2024)

Model Compression and Efficient Inference for Large Language Models: A Survey
by: Wang, Wenxiao, et al.
Published: (2024)

Radio: Rate-Distortion Optimization for Large Language Model Compression
by: Young, Sean I.
Published: (2025)

Extreme Compression of Large Language Models via Additive Quantization
by: Egiazarian, Vage, et al.
Published: (2024)

PPC-GPT: Federated Task-Specific Compression of Large Language Models via Pruning and Chain-of-Thought Distillation
by: Fan, Tao, et al.
Published: (2025)

Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models
by: Yang, Junjie, et al.
Published: (2025)

Self-Data Distillation for Recovering Quality in Pruned Large Language Models
by: Thangarasa, Vithursan, et al.
Published: (2024)

SODA: Semi On-Policy Black-Box Distillation for Large Language Models
by: Chen, Xiwen, et al.
Published: (2026)

On the Compressibility of Quantized Large Language Models
by: Mao, Yu, et al.
Published: (2024)

KDFlow: A User-Friendly and Efficient Knowledge Distillation Framework for Large Language Models
by: Zhang, Songming, et al.
Published: (2026)

SliceGPT: Compress Large Language Models by Deleting Rows and Columns
by: Ashkboos, Saleh, et al.
Published: (2024)