Cover Image

Self-Data Distillation for Recovering Quality in Pruned Large Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Thangarasa, Vithursan, Venkatesh, Ganesh, Lasby, Mike, Sinnadurai, Nish, Lie, Sean
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

TapOut: A Bandit-Based Approach to Dynamic Speculative Decoding
by: Sridhar, Aditya, et al.
Published: (2025)

MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models
by: Ganesan, Mugilan, et al.
Published: (2025)

SD$^2$: Self-Distilled Sparse Drafters
by: Lasby, Mike, et al.
Published: (2025)

REAP the Experts: Why Pruning Prevails for One-Shot MoE compression
by: Lasby, Mike, et al.
Published: (2025)

Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency
by: Thangarasa, Vithursan, et al.
Published: (2023)

MediSwift: Efficient Sparse Pre-trained Biomedical Language Models
by: Thangarasa, Vithursan, et al.
Published: (2024)

SDMPrune: Self-Distillation MLP Pruning for Efficient Large Language Models
by: Zhu, Hourun, et al.
Published: (2025)

Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
by: Zhao, Siyan, et al.
Published: (2026)

Beware of Calibration Data for Pruning Large Language Models
by: Ji, Yixin, et al.
Published: (2024)

Compact Language Models via Pruning and Knowledge Distillation
by: Muralidharan, Saurav, et al.
Published: (2024)

Pruning and Distilling Mixture-of-Experts into Dense Language Models
by: Kim, Junhyuck, et al.
Published: (2026)

Large Language Model Pruning
by: Huang, Hanjuan, et al.
Published: (2024)

Wanda++: Pruning Large Language Models via Regional Gradients
by: Yang, Yifan, et al.
Published: (2025)

Deterministic Differentiable Structured Pruning for Large Language Models
by: Huang, Weiyu, et al.
Published: (2026)

Fast and Effective Weight Update for Pruned Large Language Models
by: Boža, Vladimír
Published: (2024)

DarwinLM: Evolutionary Structured Pruning of Large Language Models
by: Tang, Shengkun, et al.
Published: (2025)

Frustratingly Easy Task-aware Pruning for Large Language Models
by: Tian, Yuanhe, et al.
Published: (2025)

Detecting and Pruning Prominent but Detrimental Neurons in Large Language Models
by: Ali, Ameen, et al.
Published: (2025)

PIP: Perturbation-based Iterative Pruning for Large Language Models
by: Cao, Yi, et al.
Published: (2025)

PPC-GPT: Federated Task-Specific Compression of Large Language Models via Pruning and Chain-of-Thought Distillation
by: Fan, Tao, et al.
Published: (2025)

Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipeline
by: Lee, Tony, et al.
Published: (2026)

DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models
by: Gao, Shangqian, et al.
Published: (2024)

Efficient Post-Training Pruning of Large Language Models with Statistical Correction
by: Yu, Peiqi, et al.
Published: (2026)

Olica: Efficient Structured Pruning of Large Language Models without Retraining
by: He, Jiujun, et al.
Published: (2025)

Text Quality-Based Pruning for Efficient Training of Language Models
by: Sharma, Vasu, et al.
Published: (2024)

Language Model-Driven Data Pruning Enables Efficient Active Learning
by: Azeemi, Abdul Hameed, et al.
Published: (2024)

Think Before You Prune: Self-Reflective Structured Pruning for Reasoning Language Models
by: Wang, Ziyan, et al.
Published: (2025)

Radio: Rate-Distortion Optimization for Large Language Model Compression
by: Young, Sean I.
Published: (2025)

Self-Refining Language Model Anonymizers via Adversarial Distillation
by: Kim, Kyuyoung, et al.
Published: (2025)

Adversarial Moment-Matching Distillation of Large Language Models
by: Jia, Chen
Published: (2024)

A Survey of On-Policy Distillation for Large Language Models
by: Song, Mingyang, et al.
Published: (2026)

SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
by: Tang, Shengkun, et al.
Published: (2026)

Foundations of Large Language Model Compression -- Part 1: Weight Quantization
by: Young, Sean I.
Published: (2024)

Rethinking Pruning Large Language Models: Benefits and Pitfalls of Reconstruction Error Minimization
by: Shin, Sungbin, et al.
Published: (2024)

MoreauPruner: Robust Pruning of Large Language Models against Weight Perturbations
by: Wang, Zixiao, et al.
Published: (2024)

Time Will Tell: Timing Side Channels via Output Token Count in Large Language Models
by: Zhang, Tianchen, et al.
Published: (2024)

Relative Kinetic Utility for Reasoning-Aware Structural Pruning in Large Language Models
by: Qian, Tianhao
Published: (2026)

On Importance of Pruning and Distillation for Efficient Low Resource NLP
by: Mirashi, Aishwarya, et al.
Published: (2024)

Multi-Reference Preference Optimization for Large Language Models
by: Le, Hung, et al.
Published: (2024)

Self-Calibrating Language Models via Test-Time Discriminative Distillation
by: Hedna, Mohamed Rissal, et al.
Published: (2026)