Imagen de Portada

Weight-Inherited Distillation for Task-Agnostic BERT Compression

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Wu, Taiqiang, Hou, Cheng, Lao, Shanshan, Li, Jiayi, Wong, Ngai, Zhao, Zhe, Yang, Yujiu
Formato:	Preprint
Publicado:	2023
Materias:	Computation and Language Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

LoCa: Logit Calibration for Knowledge Distillation
por: Yang, Runming, et al.
Publicado: (2024)

LLM-NEO: Parameter Efficient Knowledge Distillation for Large Language Models
por: Yang, Runming, et al.
Publicado: (2024)

Edge-free but Structure-aware: Prototype-Guided Knowledge Distillation from GNNs to MLPs
por: Wu, Taiqiang, et al.
Publicado: (2023)

LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
por: Pan, Zhuoshi, et al.
Publicado: (2024)

Mixture-of-Subspaces in Low-Rank Adaptation
por: Wu, Taiqiang, et al.
Publicado: (2024)

MCUBERT: Memory-Efficient BERT Inference on Commodity Microcontrollers
por: Yang, Zebin, et al.
Publicado: (2024)

Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrast
por: Shi, Chufan, et al.
Publicado: (2024)

Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models
por: Wu, Taiqiang, et al.
Publicado: (2024)

Shadow-FT: Tuning Instruct Model via Training on Paired Base Model
por: Wu, Taiqiang, et al.
Publicado: (2025)

ROMER: Expert Replacement and Router Calibration for Robust MoE LLMs on Analog Compute-in-Memory Systems
por: Zhou, Wenyong, et al.
Publicado: (2026)

Internalize the Temperature: On-Policy Self-Distillation as Policy Reheater for Reinforcement Learning
por: Yang, Xuewei, et al.
Publicado: (2026)

Understanding and Mitigating Bias Inheritance in LLM-based Data Augmentation on Downstream Tasks
por: Li, Miaomiao, et al.
Publicado: (2025)

An investigation of structures responsible for gender bias in BERT and DistilBERT
por: Leteno, Thibaud, et al.
Publicado: (2024)

MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models
por: Zhang, Ying, et al.
Publicado: (2024)

Progressive Knowledge Graph Completion
por: Li, Jiayi, et al.
Publicado: (2024)

Selection-p: Self-Supervised Task-Agnostic Prompt Compression for Faithfulness and Transferability
por: Chung, Tsz Ting, et al.
Publicado: (2024)

MiniDisc: Minimal Distillation Schedule for Language Model Compression
por: Zhang, Chen, et al.
Publicado: (2022)

Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
por: Zhang, Zeliang, et al.
Publicado: (2024)

Exploring the Limits of Model Compression in LLMs: A Knowledge Distillation Study on QA Tasks
por: Datta, Joyeeta, et al.
Publicado: (2025)

X-Coder: Advancing Competitive Programming with Fully Synthetic Tasks, Solutions, and Tests
por: Wu, Jie, et al.
Publicado: (2026)

Goal-Directed Search Outperforms Goal-Agnostic Memory Compression in Long-Context Memory Tasks
por: Zheng, Yicong, et al.
Publicado: (2025)

CVPD at QIAS 2025 Shared Task: An Efficient Encoder-Based Approach for Islamic Inheritance Reasoning
por: Bekhouche, Salah Eddine, et al.
Publicado: (2025)

Enhancing TinyBERT for Financial Sentiment Analysis Using GPT-Augmented FinBERT Distillation
por: Thomas, Graison Jos
Publicado: (2024)

LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models
por: Yang, Yifan, et al.
Publicado: (2024)

Feature Structure Distillation with Centered Kernel Alignment in BERT Transferring
por: Jung, Hee-Jun, et al.
Publicado: (2022)

Structural Rationale Distillation via Reasoning Space Compression
por: Yang, Jialin, et al.
Publicado: (2026)

Can We Use Probing to Better Understand Fine-tuning and Knowledge Distillation of the BERT NLU?
por: Hościłowicz, Jakub, et al.
Publicado: (2023)

DE$^3$-BERT: Distance-Enhanced Early Exiting for BERT based on Prototypical Networks
por: He, Jianing, et al.
Publicado: (2024)

Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillation
por: Zhang, Hengyuan, et al.
Publicado: (2025)

PPC-GPT: Federated Task-Specific Compression of Large Language Models via Pruning and Chain-of-Thought Distillation
por: Fan, Tao, et al.
Publicado: (2025)

OmniOPD: Logit-Free On-Policy Distillation via Speculative Verification
por: Zhou, Yuhang, et al.
Publicado: (2026)

The Art of Efficient Reasoning: Data, Reward, and Optimization
por: Wu, Taiqiang, et al.
Publicado: (2026)

HMS-BERT: Hybrid Multi-Task Self-Training for Multilingual and Multi-Label Cyberbullying Detection
por: Feng, Zixin, et al.
Publicado: (2026)

SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillation
por: Li, Zichong, et al.
Publicado: (2025)

Towards Token-Level Text Anomaly Detection
por: Cao, Yang, et al.
Publicado: (2026)

TACO-RL: Task Aware Prompt Compression Optimization with Reinforcement Learning
por: Shandilya, Shivam, et al.
Publicado: (2024)

KVSculpt: KV Cache Compression as Distillation
por: Jiang, Bo, et al.
Publicado: (2026)

MaxPoolBERT: Enhancing BERT Classification via Layer- and Token-Wise Aggregation
por: Behrendt, Maike, et al.
Publicado: (2025)

Revisiting Model Interpolation for Efficient Reasoning
por: Wu, Taiqiang, et al.
Publicado: (2025)

Compressing LLMs: The Truth is Rarely Pure and Never Simple
por: Jaiswal, Ajay, et al.
Publicado: (2023)