SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillation

Documenti analoghi

• SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
  di: Tang, Shengkun, et al.
  Pubblicazione: (2026)
• Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
  di: Wu, Haoyuan, et al.
  Pubblicazione: (2025)
• GRIN: GRadient-INformed MoE
  di: Liu, Liyuan, et al.
  Pubblicazione: (2024)
• Leave It to the Experts: Detecting Knowledge Distillation via MoE Expert Signatures
  di: Li, Pingzhi, et al.
  Pubblicazione: (2025)
• Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation
  di: Li, Zichong, et al.
  Pubblicazione: (2026)