Copertina

Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Wu, Taiqiang, Tao, Chaofan, Wang, Jiahao, Yang, Runming, Zhao, Zhe, Wong, Ngai
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Computation and Language Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

LLM-NEO: Parameter Efficient Knowledge Distillation for Large Language Models
di: Yang, Runming, et al.
Pubblicazione: (2024)

Revisiting Model Interpolation for Efficient Reasoning
di: Wu, Taiqiang, et al.
Pubblicazione: (2025)

Timber: Training-free Instruct Model Refining with Base via Effective Rank
di: Wu, Taiqiang, et al.
Pubblicazione: (2025)

Better Estimation of the Kullback--Leibler Divergence Between Language Models
di: Amini, Afra, et al.
Pubblicazione: (2025)

Vocabulary Expansion of Large Language Models via Kullback-Leibler-Based Self-Distillation
di: Linder, Max Rehman
Pubblicazione: (2025)

Shadow-FT: Tuning Instruct Model via Training on Paired Base Model
di: Wu, Taiqiang, et al.
Pubblicazione: (2025)

Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation
di: Lv, Jiaming, et al.
Pubblicazione: (2024)

Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation
di: Luong, Hoang-Chau, et al.
Pubblicazione: (2026)

The Art of Efficient Reasoning: Data, Reward, and Optimization
di: Wu, Taiqiang, et al.
Pubblicazione: (2026)

ProFit: Leveraging High-Value Signals in SFT via Probability-Guided Token Selection
di: Liu, Tao, et al.
Pubblicazione: (2026)

LoCa: Logit Calibration for Knowledge Distillation
di: Yang, Runming, et al.
Pubblicazione: (2024)

Edge-free but Structure-aware: Prototype-Guided Knowledge Distillation from GNNs to MLPs
di: Wu, Taiqiang, et al.
Pubblicazione: (2023)

Generalized Kullback-Leibler Divergence Loss
di: Cui, Jiequan, et al.
Pubblicazione: (2025)

Mixture-of-Subspaces in Low-Rank Adaptation
di: Wu, Taiqiang, et al.
Pubblicazione: (2024)

A Survey on the Honesty of Large Language Models
di: Li, Siheng, et al.
Pubblicazione: (2024)

Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning
di: Li, Zhen, et al.
Pubblicazione: (2025)

Entropy and the Kullback-Leibler Divergence for Bayesian Networks: Computational Complexity and Efficient Implementation
di: Scutari, Marco
Pubblicazione: (2023)

EnviroExam: Benchmarking Environmental Science Knowledge of Large Language Models
di: Huang, Yu, et al.
Pubblicazione: (2024)

Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies
di: Tao, Chaofan, et al.
Pubblicazione: (2024)

Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application
di: Yang, Chuanpeng, et al.
Pubblicazione: (2024)

Establishing a Scale for Kullback-Leibler Divergence in Language Models Across Various Settings
di: Kishino, Ryo, et al.
Pubblicazione: (2025)

Cross-Modal Knowledge Distillation for Speech Large Language Models
di: Wang, Enzhi, et al.
Pubblicazione: (2025)

EasyDistill: A Comprehensive Toolkit for Effective Knowledge Distillation of Large Language Models
di: Wang, Chengyu, et al.
Pubblicazione: (2025)

LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models
di: Yang, Yifan, et al.
Pubblicazione: (2024)

Dual-Space Knowledge Distillation for Large Language Models
di: Zhang, Songming, et al.
Pubblicazione: (2024)

The Labyrinth and the Thread: Rethinking Regularizations in Sequential Knowledge Editing for Large Language Models
di: Wang, Zheng, et al.
Pubblicazione: (2026)

Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Models
di: Amayuelas, Alfonso, et al.
Pubblicazione: (2023)

Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
di: Li, Yaxuan, et al.
Pubblicazione: (2026)

Distilling Event Sequence Knowledge From Large Language Models
di: Wadhwa, Somin, et al.
Pubblicazione: (2024)

Rethinking the Potential of Multimodality in Collaborative Problem Solving Diagnosis with Large Language Models
di: Wong, K., et al.
Pubblicazione: (2025)

Delta Knowledge Distillation for Large Language Models
di: Cao, Yihan, et al.
Pubblicazione: (2025)

The Knowledge Alignment Problem: Bridging Human and External Knowledge for Large Language Models
di: Zhang, Shuo, et al.
Pubblicazione: (2023)

LLMR: Knowledge Distillation with a Large Language Model-Induced Reward
di: Li, Dongheng, et al.
Pubblicazione: (2024)

Contextualization Distillation from Large Language Model for Knowledge Graph Completion
di: Li, Dawei, et al.
Pubblicazione: (2024)

Knowledge Distillation for Large Language Models
di: La Torre, Alejandro Paredes, et al.
Pubblicazione: (2026)

Weight-Inherited Distillation for Task-Agnostic BERT Compression
di: Wu, Taiqiang, et al.
Pubblicazione: (2023)

Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling
di: Yang, Linyao, et al.
Pubblicazione: (2023)

Divergent Creativity in Humans and Large Language Models
di: Bellemare-Pepin, Antoine, et al.
Pubblicazione: (2024)

FedCoT: Federated Chain-of-Thought Distillation for Large Language Models
di: Fan, Tao, et al.
Pubblicazione: (2024)

ELPO: Ensemble Learning Based Prompt Optimization for Large Language Models
di: Zhang, Qing, et al.
Pubblicazione: (2025)