Cover Image

MiniPLM: Knowledge Distillation for Pre-Training Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Gu, Yuxian, Zhou, Hao, Meng, Fandong, Zhou, Jie, Huang, Minlie
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

MiniLLM: On-Policy Distillation of Large Language Models
by: Gu, Yuxian, et al.
Published: (2023)

Large Language Models Are Not Robust Multiple Choice Selectors
by: Zheng, Chujie, et al.
Published: (2023)

Instruction Pre-Training: Language Models are Supervised Multitask Learners
by: Cheng, Daixuan, et al.
Published: (2024)

Beyond Next Token Prediction: Patch-Level Training for Large Language Models
by: Shao, Chenze, et al.
Published: (2024)

Towards Optimal Learning of Language Models
by: Gu, Yuxian, et al.
Published: (2024)

On Prompt-Driven Safeguarding for Large Language Models
by: Zheng, Chujie, et al.
Published: (2024)

Warmup-Distill: Bridge the Distribution Mismatch between Teacher and Student before Knowledge Distillation
by: Sun, Zengkui, et al.
Published: (2025)

Accelerating Inference in Large Language Models with a Unified Layer Skipping Strategy
by: Liu, Yijin, et al.
Published: (2024)

Enhancing Cross-Tokenizer Knowledge Distillation with Contextual Dynamical Mapping
by: Chen, Yijie, et al.
Published: (2025)

A Dual-Space Framework for General Knowledge Distillation of Large Language Models
by: Zhang, Xue, et al.
Published: (2025)

Data Selection via Optimal Control for Language Models
by: Gu, Yuxian, et al.
Published: (2024)

Direct Preference Knowledge Distillation for Large Language Models
by: Li, Yixing, et al.
Published: (2024)

TIM: Teaching Large Language Models to Translate with Comparison
by: Zeng, Jiali, et al.
Published: (2023)

Improving Machine Translation with Large Language Models: A Preliminary Study with Cooperative Decoding
by: Zeng, Jiali, et al.
Published: (2023)

Generative Multi-Modal Knowledge Retrieval with Large Language Models
by: Long, Xinwei, et al.
Published: (2024)

Retrieval-Augmented Machine Translation with Unstructured Knowledge
by: Wang, Jiaan, et al.
Published: (2024)

TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models
by: Yang, Zhen, et al.
Published: (2023)

Continuous Autoregressive Language Models
by: Shao, Chenze, et al.
Published: (2025)

The Missing Half: Unveiling Training-time Implicit Safety Risks Beyond Deployment
by: Zhang, Zhexin, et al.
Published: (2026)

THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translation
by: Liang, Yunlong, et al.
Published: (2025)

Figure It Out: Improve the Frontier of Reasoning with Executable Visual States
by: Chen, Meiqi, et al.
Published: (2025)

Multilingual Knowledge Editing with Language-Agnostic Factual Neurons
by: Zhang, Xue, et al.
Published: (2024)

Language Generation with Strictly Proper Scoring Rules
by: Shao, Chenze, et al.
Published: (2024)

CRAT: A Multi-Agent Framework for Causality-Enhanced Reflective and Retrieval-Augmented Translation with Large Language Models
by: Chen, Meiqi, et al.
Published: (2024)

CSCD-NS: a Chinese Spelling Check Dataset for Native Speakers
by: Hu, Yong, et al.
Published: (2022)

DeepTrans: Deep Reasoning Translation via Reinforcement Learning
by: Wang, Jiaan, et al.
Published: (2025)

ExTrans: Multilingual Deep Reasoning Translation via Exemplar-Enhanced Reinforcement Learning
by: Wang, Jiaan, et al.
Published: (2025)

Distilling Rule-based Knowledge into Large Language Models
by: Yang, Wenkai, et al.
Published: (2023)

Unsupervised Information Refinement Training of Large Language Models for Retrieval-Augmented Generation
by: Xu, Shicheng, et al.
Published: (2024)

An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
by: Luo, Yun, et al.
Published: (2023)

Understanding and Addressing the Under-Translation Problem from the Perspective of Decoding Objective
by: Shao, Chenze, et al.
Published: (2024)

SlangDIT: Benchmarking LLMs in Interpretative Slang Translation
by: Liang, Yunlong, et al.
Published: (2025)

Cross-layer Attention Sharing for Pre-trained Large Language Models
by: Mu, Yongyu, et al.
Published: (2024)

LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learning
by: Lan, Zhibin, et al.
Published: (2025)

TasTe: Teaching Large Language Models to Translate through Self-Reflection
by: Wang, Yutong, et al.
Published: (2024)

CM-Align: Consistency-based Multilingual Alignment for Large Language Models
by: Zhang, Xue, et al.
Published: (2025)

RadarPLM: Adapting Pre-trained Language Models for Marine Radar Target Detection by Selective Fine-tuning
by: Hu, Qiying, et al.
Published: (2025)

DRT: Deep Reasoning Translation via Long Chain-of-Thought
by: Wang, Jiaan, et al.
Published: (2024)

EAG: Extract and Generate Multi-way Aligned Corpus for Complete Multi-lingual Neural Machine Translation
by: Xu, Yulin, et al.
Published: (2022)

Knowledge-Driven Agentic Scientific Corpus Distillation Framework for Biomedical Large Language Models Training
by: Xiao, Meng, et al.
Published: (2025)