Cover Image

Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Fang, Luyang, Yu, Xiaowei, Cai, Jiazhang, Chen, Yongkai, Wu, Shushan, Liu, Zhengliang, Yang, Zhenyuan, Lu, Haoran, Gong, Xilin, Liu, Yufang, Ma, Terry, Ruan, Wei, Abbasi, Ali, Zhang, Jing, Wang, Tao, Latif, Ehsan, You, Weihang, Jiang, Hanqi, Liu, Wei, Zhang, Wei, Kolouri, Soheil, Zhai, Xiaoming, Zhu, Dajiang, Zhong, Wenxuan, Liu, Tianming, Ma, Ping
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Multi-Teacher Knowledge Distillation via Teacher-Informed Mixture Priors
by: Fang, Luyang, et al.
Published: (2026)

Knowledge Distillation of LLM for Automatic Scoring of Science Education Assessments
by: Latif, Ehsan, et al.
Published: (2023)

Vector-Quantized Soft Label Compression for Dataset Distillation
by: Abbasi, Ali, et al.
Published: (2026)

DCMM-Transformer: Degree-Corrected Mixed-Membership Attention for Medical Imaging
by: Cheng, Huimin, et al.
Published: (2025)

One Category One Prompt: Dataset Distillation using Diffusion Models
by: Abbasi, Ali, et al.
Published: (2024)

S2MNet: Speckle-To-Mesh Net for Three-Dimensional Cardiac Morphology Reconstruction via Echocardiogram
by: Gong, Xilin, et al.
Published: (2025)

Foundation Models for Low-Resource Language Education (Vision Paper)
by: Ding, Zhaojun, et al.
Published: (2024)

Thinking with Gaze: Sequential Eye-Tracking as Visual Reasoning Supervision for Medical VLMs
by: Li, Yiwei, et al.
Published: (2026)

Robust Core-Periphery Constrained Transformer for Domain Adaptation
by: Yu, Xiaowei, et al.
Published: (2023)

ConQuR: Corner Aligned Activation Quantization via Optimized Rotations for LLMs
by: Thrash, Chayne, et al.
Published: (2026)

Diffusion-Augmented Coreset Expansion for Scalable Dataset Distillation
by: Abbasi, Ali, et al.
Published: (2024)

Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning
by: Ma, Chong, et al.
Published: (2024)

Generalizable and Efficient Automated Scoring with a Knowledge-Distilled Multi-Task Mixture-of-Experts
by: Fang, Luyang, et al.
Published: (2025)

AI Gender Bias, Disparities, and Fairness: Does Training Data Matter?
by: Latif, Ehsan, et al.
Published: (2023)

Advancing Education through Tutoring Systems: A Systematic Literature Review
by: Liu, Vincent, et al.
Published: (2025)

MolQAE: Quantum Autoencoder for Molecular Representation Learning
by: Pan, Yi, et al.
Published: (2025)

Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Research
by: Zhong, Tianyang, et al.
Published: (2024)

Efficient Multi-Task Inferencing: Model Merging with Gromov-Wasserstein Feature Alignment
by: Fang, Luyang, et al.
Published: (2025)

EMPEROR: Efficient Moment-Preserving Representation of Distributions
by: Liu, Xinran, et al.
Published: (2025)

Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
by: Lu, Haoran, et al.
Published: (2025)

Achieving Fine-grained Cross-modal Understanding through Brain-inspired Hierarchical Representation Learning
by: You, Weihang, et al.
Published: (2026)

ADLGen: Synthesizing Symbolic, Event-Triggered Sensor Sequences for Human Activity Modeling
by: You, Weihang, et al.
Published: (2025)

Large Language Models for Assisting American College Applications
by: Liu, Zhengliang, et al.
Published: (2026)

A recent evaluation on the performance of LLMs on radiation oncology physics using questions of randomly shuffled options
by: Wang, Peilong, et al.
Published: (2024)

Min Generalized Sliced Gromov Wasserstein: A Scalable Path to Gromov Wasserstein
by: Shahbazi, Ashkan, et al.
Published: (2026)

Privacy-Preserved Automated Scoring using Federated Learning for Educational Research
by: Latif, Ehsan, et al.
Published: (2025)

Efficient Multi-Task Inferencing with a Shared Backbone and Lightweight Task-Specific Adapters for Automatic Scoring
by: Latif, Ehsan, et al.
Published: (2024)

NoisyNN: Exploring the Impact of Information Entropy Change in Learning Systems
by: Yu, Xiaowei, et al.
Published: (2023)

Distillation-Enhanced Physical Adversarial Attacks
by: Liu, Wei, et al.
Published: (2025)

Wahkon: A Statistically Principled Deep RKHS Superposition Network
by: Chen, Yongkai, et al.
Published: (2026)

Quantum Statistical Bootstrap
by: Chen, Yongkai, et al.
Published: (2026)

Low-Rank Prehab: Preparing Neural Networks for SVD Compression
by: Qin, Haoran, et al.
Published: (2025)

IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression
by: Abbasi, Ali, et al.
Published: (2026)

A Systematic Assessment of OpenAI o1-Preview for Higher Order Thinking in Education
by: Latif, Ehsan, et al.
Published: (2024)

JDCNet: Confidence-Gated Privileged-Modality Distillation for Cost-Preserving X-ray Inference
by: Ma, Bo, et al.
Published: (2026)

Constrained Sliced Wasserstein Embedding
by: NaderiAlizadeh, Navid, et al.
Published: (2025)

EG-SpikeFormer: Eye-Gaze Guided Transformer on Spiking Neural Networks for Medical Image Analysis
by: Pan, Yi, et al.
Published: (2024)

AGI: Artificial General Intelligence for Education
by: Latif, Ehsan, et al.
Published: (2023)

LLM-POTUS Score: A Framework of Analyzing Presidential Debates with Large Language Models
by: Liu, Zhengliang, et al.
Published: (2024)

NOLA: Compressing LoRA using Linear Combination of Random Basis
by: Koohpayegani, Soroush Abbasi, et al.
Published: (2023)