Image de couverture de livre

MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Zhang, Ying, Yang, Ziheng, Ji, Shufan
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Efficient Knowledge Probing of Large Language Models by Adapting Pre-trained Embeddings
par: Sharma, Kartik, et autres
Publié: (2025)

Chinese MentalBERT: Domain-Adaptive Pre-training on Social Media for Chinese Mental Health Text Analysis
par: Zhai, Wei, et autres
Publié: (2024)

Model Merging in Pre-training of Large Language Models
par: Li, Yunshui, et autres
Publié: (2025)

An investigation of structures responsible for gender bias in BERT and DistilBERT
par: Leteno, Thibaud, et autres
Publié: (2024)

Weight-Inherited Distillation for Task-Agnostic BERT Compression
par: Wu, Taiqiang, et autres
Publié: (2023)

Pre-training Limited Memory Language Models with Internal and External Knowledge
par: Zhao, Linxi, et autres
Publié: (2025)

Can We Use Probing to Better Understand Fine-tuning and Knowledge Distillation of the BERT NLU?
par: Hościłowicz, Jakub, et autres
Publié: (2023)

DEPT: Decoupled Embeddings for Pre-training Language Models
par: Iacob, Alex, et autres
Publié: (2024)

Comparative Study of Pre-Trained BERT and Large Language Models for Code-Mixed Named Entity Recognition
par: Shirke, Mayur, et autres
Publié: (2025)

Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
par: Fang, Luyang, et autres
Publié: (2025)

Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models
par: Yang, Junjie, et autres
Publié: (2025)

Benchmarking BERT-based Models for Sentence-level Topic Classification in Nepali Language
par: Karki, Nischal, et autres
Publié: (2026)

Making Pre-trained Language Models Great on Tabular Prediction
par: Yan, Jiahuan, et autres
Publié: (2024)

Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training
par: Chung, Woojin, et autres
Publié: (2025)

SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
par: Tang, Shengkun, et autres
Publié: (2026)

MediSwift: Efficient Sparse Pre-trained Biomedical Language Models
par: Thangarasa, Vithursan, et autres
Publié: (2024)

PaPaformer: Language Model from Pre-trained Parallel Paths
par: Tapaninaho, Joonas, et autres
Publié: (2025)

Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Models
par: Zheng, Junhao, et autres
Publié: (2023)

Does Pre-trained Language Model Actually Infer Unseen Links in Knowledge Graph Completion?
par: Sakai, Yusuke, et autres
Publié: (2023)

CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Models
par: Gu, Jiawei, et autres
Publié: (2024)

Knowledge Distillation from Large Language Models for Household Energy Modeling
par: Takrouri, Mohannad, et autres
Publié: (2025)

A Survey on Symbolic Knowledge Distillation of Large Language Models
par: Acharya, Kamal, et autres
Publié: (2024)

Edit Less, Achieve More: Dynamic Sparse Neuron Masking for Lifelong Knowledge Editing in LLMs
par: Liu, Jinzhe, et autres
Publié: (2025)

Language Model Knowledge Distillation for Efficient Question Answering in Spanish
par: Bazaga, Adrián, et autres
Publié: (2023)

Thinking Augmented Pre-training
par: Wang, Liang, et autres
Publié: (2025)

COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training
par: Kim, Sanghwan, et autres
Publié: (2024)

DiLM: Distilling Dataset into Language Model for Text-level Dataset Distillation
par: Maekawa, Aru, et autres
Publié: (2024)

GottBERT: a pure German Language Model
par: Scheible, Raphael, et autres
Publié: (2020)

Fine-Tuning Pre-trained Language Models to Detect In-Game Trash Talks
par: Fesalbon, Daniel, et autres
Publié: (2024)

Pre-trained Large Language Models Use Fourier Features to Compute Addition
par: Zhou, Tianyi, et autres
Publié: (2024)

HLAT: High-quality Large Language Model Pre-trained on AWS Trainium
par: Fan, Haozheng, et autres
Publié: (2024)

Structural Pruning of Pre-trained Language Models via Neural Architecture Search
par: Klein, Aaron, et autres
Publié: (2024)

Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training
par: Zhong, Zexuan, et autres
Publié: (2024)

Hadamard Adapter: An Extreme Parameter-Efficient Adapter Tuning Method for Pre-trained Language Models
par: Chen, Yuyan, et autres
Publié: (2024)

Trust Region On-Policy Distillation
par: Xing, Xingrun, et autres
Publié: (2026)

A Contextualized BERT model for Knowledge Graph Completion
par: Gul, Haji, et autres
Publié: (2024)

Investigating Data Contamination for Pre-training Language Models
par: Jiang, Minhao, et autres
Publié: (2024)

Aligning Pre-trained Models for Spoken Language Translation
par: Sedláček, Šimon, et autres
Publié: (2024)

Sequence-to-Sequence Spanish Pre-trained Language Models
par: Araujo, Vladimir, et autres
Publié: (2023)

Sparse is Enough in Fine-tuning Pre-trained Large Language Models
par: Song, Weixi, et autres
Publié: (2023)