Cover Image

LLM-NEO: Parameter Efficient Knowledge Distillation for Large Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Yang, Runming, Wu, Taiqiang, Wang, Jiahao, Hu, Pengfei, Wu, Yik-Chung, Wong, Ngai, Yang, Yujiu
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Shadow-FT: Tuning Instruct Model via Training on Paired Base Model
by: Wu, Taiqiang, et al.
Published: (2025)

Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models
by: Wu, Taiqiang, et al.
Published: (2024)

Revisiting Model Interpolation for Efficient Reasoning
by: Wu, Taiqiang, et al.
Published: (2025)

LoCa: Logit Calibration for Knowledge Distillation
by: Yang, Runming, et al.
Published: (2024)

Timber: Training-free Instruct Model Refining with Base via Effective Rank
by: Wu, Taiqiang, et al.
Published: (2025)

Edge-free but Structure-aware: Prototype-Guided Knowledge Distillation from GNNs to MLPs
by: Wu, Taiqiang, et al.
Published: (2023)

ProFit: Leveraging High-Value Signals in SFT via Probability-Guided Token Selection
by: Liu, Tao, et al.
Published: (2026)

The Art of Efficient Reasoning: Data, Reward, and Optimization
by: Wu, Taiqiang, et al.
Published: (2026)

Mixture-of-Subspaces in Low-Rank Adaptation
by: Wu, Taiqiang, et al.
Published: (2024)

LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models
by: Yang, Yifan, et al.
Published: (2024)

A Survey on the Honesty of Large Language Models
by: Li, Siheng, et al.
Published: (2024)

Weight-Inherited Distillation for Task-Agnostic BERT Compression
by: Wu, Taiqiang, et al.
Published: (2023)

LLM-Barber: Block-Aware Rebuilder for Sparsity Mask in One-Shot for Large Language Models
by: Su, Yupeng, et al.
Published: (2024)

KDFlow: A User-Friendly and Efficient Knowledge Distillation Framework for Large Language Models
by: Zhang, Songming, et al.
Published: (2026)

PTQTP: Post-Training Quantization to Trit-Planes for Large Language Models
by: Xiao, He, et al.
Published: (2025)

ELPO: Ensemble Learning Based Prompt Optimization for Large Language Models
by: Zhang, Qing, et al.
Published: (2025)

Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning
by: Li, Zhen, et al.
Published: (2025)

Distributional Clarity: The Hidden Driver of RL-Friendliness in Large Language Models
by: Sun, Shaoning, et al.
Published: (2026)

Delta Knowledge Distillation for Large Language Models
by: Cao, Yihan, et al.
Published: (2025)

MCUBERT: Memory-Efficient BERT Inference on Commodity Microcontrollers
by: Yang, Zebin, et al.
Published: (2024)

LLM Surgery: Efficient Knowledge Unlearning and Editing in Large Language Models
by: Veldanda, Akshaj Kumar, et al.
Published: (2024)

Mitigating Reversal Curse in Large Language Models via Semantic-aware Permutation Training
by: Guo, Qingyan, et al.
Published: (2024)

APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Models
by: Guan, Ziyi, et al.
Published: (2024)

LLM2: Let Large Language Models Harness System 2 Reasoning
by: Yang, Cheng, et al.
Published: (2024)

Progressive Knowledge Graph Completion
by: Li, Jiayi, et al.
Published: (2024)

DistiLLM: Towards Streamlined Distillation for Large Language Models
by: Ko, Jongwoo, et al.
Published: (2024)

Parameter Efficient Diverse Paraphrase Generation Using Sequence-Level Knowledge Distillation
by: Jayawardena, Lasal, et al.
Published: (2024)

Parameter Efficient Instruction Tuning: An Empirical Study
by: He, Pengfei
Published: (2024)

Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models
by: Kim, Gyeongman, et al.
Published: (2025)

FactorLLM: Factorizing Knowledge via Mixture of Experts for Large Language Models
by: Zhao, Zhongyu, et al.
Published: (2024)

I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models
by: Hu, Xing, et al.
Published: (2024)

Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies
by: Wang, Luping, et al.
Published: (2024)

TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models
by: Shing, Makoto, et al.
Published: (2025)

zrLLM: Zero-Shot Relational Learning on Temporal Knowledge Graphs with Large Language Models
by: Ding, Zifeng, et al.
Published: (2023)

A Dual-Space Framework for General Knowledge Distillation of Large Language Models
by: Zhang, Xue, et al.
Published: (2025)

Structured Agent Distillation for Large Language Model
by: Liu, Jun, et al.
Published: (2025)

PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning
by: Kim, Gyeongman, et al.
Published: (2024)

SDMPrune: Self-Distillation MLP Pruning for Efficient Large Language Models
by: Zhu, Hourun, et al.
Published: (2025)

EfficientLLM: Efficiency in Large Language Models
by: Yuan, Zhengqing, et al.
Published: (2025)

Rethinking the Potential of Multimodality in Collaborative Problem Solving Diagnosis with Large Language Models
by: Wong, K., et al.
Published: (2025)