Cover Image

Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Rao, Jun, Liu, Xuebo, Lin, Zepeng, Ding, Liang, Li, Jing, Tao, Dacheng, Zhang, Min
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Revisiting Knowledge Distillation for Autoregressive Language Models
by: Zhong, Qihuang, et al.
Published: (2024)

Learning from Imperfect Data: Towards Efficient Knowledge Distillation of Autoregressive Language Models for Text-to-SQL
by: Zhong, Qihuang, et al.
Published: (2024)

PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptation
by: Zhong, Qihuang, et al.
Published: (2022)

SeaPO: Strategic Error Amplification for Robust Preference Optimization of Large Language Models
by: Rao, Jun, et al.
Published: (2025)

Self-Powered LLM Modality Expansion for Large Speech-Text Models
by: Yu, Tengfei, et al.
Published: (2024)

Runaway is Ashamed, But Helpful: On the Early-Exit Behavior of Large Language Model-based Agents in Embodied Environments
by: Lu, Qingyu, et al.
Published: (2025)

A Survey on Knowledge Distillation of Large Language Models
by: Xu, Xiaohan, et al.
Published: (2024)

Revisiting Demonstration Selection Strategies in In-Context Learning
by: Peng, Keqin, et al.
Published: (2024)

DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization
by: Deng, Hexuan, et al.
Published: (2024)

APT: Improving Specialist LLM Performance with Weakness Case Acquisition and Iterative Preference Training
by: Rao, Jun, et al.
Published: (2025)

Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editing
by: Xiao, Yisong, et al.
Published: (2025)

Dynamic Sampling that Adapts: Self-Aware Iterative Data Persistent Optimization for Mathematical Reasoning
by: Rao, Jun, et al.
Published: (2025)

CommonIT: Commonality-Aware Instruction Tuning for Large Language Models via Data Partitions
by: Rao, Jun, et al.
Published: (2024)

Reason-KE++: Aligning the Process, Not Just the Outcome, for Faithful LLM Knowledge Editing
by: Wu, Yuchen, et al.
Published: (2025)

Robust Knowledge Editing via Explicit Reasoning Chains for Distractor-Resilient Multi-Hop QA
by: Wu, Yuchen, et al.
Published: (2025)

E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generation
by: Zhong, Qihuang, et al.
Published: (2022)

Uncertainty Aware Learning for Language Model Alignment
by: Wang, Yikun, et al.
Published: (2024)

Revisiting Catastrophic Forgetting in Large Language Model Tuning
by: Li, Hongyu, et al.
Published: (2024)

The Bitter Lesson of Diffusion Language Models for Agentic Workflows: A Comprehensive Reality Check
by: Lu, Qingyu, et al.
Published: (2026)

Edit Once, Update Everywhere: A Simple Framework for Cross-Lingual Knowledge Synchronization in LLMs
by: Wu, Yuchen, et al.
Published: (2025)

REA-RL: Reflection-Aware Online Reinforcement Learning for Efficient Reasoning
by: Deng, Hexuan, et al.
Published: (2025)

ROSE Doesn't Do That: Boosting the Safety of Instruction-Tuned Large Language Models with Reverse Prompt Contrastive Decoding
by: Zhong, Qihuang, et al.
Published: (2024)

Knowledge Editing with Dynamic Knowledge Graphs for Multi-Hop Question Answering
by: Lu, Yifan, et al.
Published: (2024)

Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models
by: Yang, Junjie, et al.
Published: (2025)

Enhancing Knowledge Distillation of Large Language Models through Efficient Multi-Modal Distribution Alignment
by: Peng, Tianyu, et al.
Published: (2024)

RouterKGQA: Specialized--General Model Routing for Constraint-Aware Knowledge Graph Question Answering
by: Yuan, Bo, et al.
Published: (2026)

OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models
by: Wang, Shuai, et al.
Published: (2024)

Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models
by: Lu, Qingyu, et al.
Published: (2023)

Entropy-Guided Watermarking for LLMs: A Test-Time Framework for Robust and Traceable Text Generation
by: Cai, Shizhan, et al.
Published: (2025)

WisdoM: Improving Multimodal Sentiment Analysis by Fusing Contextual World Knowledge
by: Wang, Wenbin, et al.
Published: (2024)

Intention Analysis Makes LLMs A Good Jailbreak Defender
by: Zhang, Yuqi, et al.
Published: (2024)

Improving Recall of Large Language Models: A Model Collaboration Approach for Relational Triple Extraction
by: Ding, Zepeng, et al.
Published: (2024)

Resolving Knowledge Conflicts in Domain-specific Data Selection: A Case Study on Medical Instruction-tuning
by: Zhong, Qihuang, et al.
Published: (2025)

KaFT: Knowledge-aware Fine-tuning for Boosting LLMs' Domain-specific Question-Answering Performance
by: Zhong, Qihuang, et al.
Published: (2025)

Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMs
by: Jin, Ruihan, et al.
Published: (2026)

NewTerm: Benchmarking Real-Time New Terms for Large Language Models with Annual Updates
by: Deng, Hexuan, et al.
Published: (2024)

Flipping Knowledge Distillation: Leveraging Small Models' Expertise to Enhance LLMs in Text Matching
by: Li, Mingzhe, et al.
Published: (2025)

DB-LLM: Accurate Dual-Binarization for Efficient LLMs
by: Chen, Hong, et al.
Published: (2024)

Knowledge Distillation for Temporal Knowledge Graph Reasoning with Large Language Models
by: Xing, Wang, et al.
Published: (2026)

Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
by: Liu, Boan, et al.
Published: (2023)