Copertina

Self-Distillation as a Performance Recovery Mechanism for LLMs: Counteracting Compression and Catastrophic Forgetting

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Liu, Chi, Chen, Xin, Zhou, Xu, Tu, Fangbo, Manoharan, Srinivasan
Natura:	Preprint
Pubblicazione:	2026
Soggetti:	Machine Learning Artificial Intelligence Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Evolutionary Strategies lead to Catastrophic Forgetting in LLMs
di: Abdi, Immanuel, et al.
Pubblicazione: (2026)

Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuning
di: Ren, Weijieying, et al.
Pubblicazione: (2024)

Intelligent Learning Rate Distribution to reduce Catastrophic Forgetting in Transformers
di: Kenneweg, Philip, et al.
Pubblicazione: (2024)

Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptation
di: Li, Junzhuo, et al.
Pubblicazione: (2025)

CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigation
di: Fawi, Muhammad
Pubblicazione: (2024)

Catastrophic Forgetting in Kolmogorov-Arnold Networks
di: Rahman, Mohammad Marufur, et al.
Pubblicazione: (2025)

Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models
di: Hsiao, Chi-Yuan, et al.
Pubblicazione: (2025)

Wings: Learning Multimodal LLMs without Text-only Forgetting
di: Zhang, Yi-Kai, et al.
Pubblicazione: (2024)

Can LLMs Alleviate Catastrophic Forgetting in Graph Continual Learning? A Systematic Study
di: Cheng, Ziyang, et al.
Pubblicazione: (2025)

Graceful Forgetting in Generative Language Models
di: Jiang, Chunyang, et al.
Pubblicazione: (2025)

Confidence-aware Self-Semantic Distillation on Knowledge Graph Embedding
di: Liu, Yichen, et al.
Pubblicazione: (2022)

Dynamic Orthogonal Continual Fine-tuning for Mitigating Catastrophic Forgettings
di: Zhang, Zhixin, et al.
Pubblicazione: (2025)

Forgetting Transformer: Softmax Attention with a Forget Gate
di: Lin, Zhixuan, et al.
Pubblicazione: (2025)

Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compression
di: Zhou, Longsheng, et al.
Pubblicazione: (2026)

Structural Rationale Distillation via Reasoning Space Compression
di: Yang, Jialin, et al.
Pubblicazione: (2026)

Forget What You Know about LLMs Evaluations -- LLMs are Like a Chameleon
di: Cohen-Inger, Nurit, et al.
Pubblicazione: (2025)

Stuffed Mamba: Oversized States Lead to the Inability to Forget
di: Chen, Yingfa, et al.
Pubblicazione: (2024)

Learning is Forgetting: LLM Training As Lossy Compression
di: Conklin, Henry C., et al.
Pubblicazione: (2026)

Sequencing to Mitigate Catastrophic Forgetting in Continual Learning
di: Moussa, Hesham G., et al.
Pubblicazione: (2025)

KVSculpt: KV Cache Compression as Distillation
di: Jiang, Bo, et al.
Pubblicazione: (2026)

Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs
di: Bhattacharyya, Sree, et al.
Pubblicazione: (2026)

Overcoming Catastrophic Forgetting by Exemplar Selection in Task-oriented Dialogue System
di: Chen, Chen, et al.
Pubblicazione: (2024)

SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales
di: Xu, Tianyang, et al.
Pubblicazione: (2024)

Probing to Refine: Reinforcement Distillation of LLMs via Explanatory Inversion
di: Tan, Zhen, et al.
Pubblicazione: (2026)

Counteracting Matthew Effect in Self-Improvement of LVLMs through Head-Tail Re-balancing
di: Guo, Xin, et al.
Pubblicazione: (2025)

Adaptive-Boundary-Clipping GRPO: Ensuring Bounded Ratios for Stable and Generalizable Training
di: Liu, Chi, et al.
Pubblicazione: (2026)

Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning
di: Elsayed, Mohamed, et al.
Pubblicazione: (2024)

Revisiting Catastrophic Forgetting in Continual Knowledge Graph Embedding
di: Pons, Gerard, et al.
Pubblicazione: (2026)

Quantifying Catastrophic Forgetting in IoT Intrusion Detection Systems
di: Banerjee, Sourasekhar, et al.
Pubblicazione: (2026)

A Conformal Predictive Measure for Assessing Catastrophic Forgetting
di: Pitsiorlas, Ioannis, et al.
Pubblicazione: (2025)

On the Implicit Adversariality of Catastrophic Forgetting in Deep Continual Learning
di: Peng, Ze, et al.
Pubblicazione: (2025)

Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents
di: Wang, Hao, et al.
Pubblicazione: (2026)

Adaptive Computation Pruning for the Forgetting Transformer
di: Lin, Zhixuan, et al.
Pubblicazione: (2025)

Distillation Contrastive Decoding: Improving LLMs Reasoning with Contrastive Decoding and Distillation
di: Phan, Phuc, et al.
Pubblicazione: (2024)

Efficiently Distilling LLMs for Edge Applications
di: Kundu, Achintya, et al.
Pubblicazione: (2024)

Scalable LLM Reasoning Acceleration with Low-rank Distillation
di: Dong, Harry, et al.
Pubblicazione: (2025)

Multilingual Safety Alignment via Self-Distillation
di: Qin, Ruiyang, et al.
Pubblicazione: (2026)

Explaining Robustness to Catastrophic Forgetting Through Incremental Concept Formation
di: Barari, Nicki, et al.
Pubblicazione: (2025)

Catastrophic Forgetting Mitigation Through Plateau Phase Activity Profiling
di: Mashiach, Idan, et al.
Pubblicazione: (2025)

SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignment
di: Huang, Yuqing, et al.
Pubblicazione: (2025)