MaskMoE: Boosting Token-Level Learning via Routing Mask in Mixture-of-Experts

Documenti analoghi

• CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Experts
  di: Su, Zhenpeng, et al.
  Pubblicazione: (2024)
• Dial-MAE: ConTextual Masked Auto-Encoder for Retrieval-based Dialogue Systems
  di: Su, Zhenpeng, et al.
  Pubblicazione: (2023)
• MiLe Loss: a New Entropy-Weighed Loss for Mitigating the Bias of Learning Difficulties in Large Language Models
  di: Su, Zhenpeng, et al.
  Pubblicazione: (2023)
• Scaffold-BPE: Enhancing Byte Pair Encoding for Large Language Models with Simple and Effective Scaffold Token Removal
  di: Lian, Haoran, et al.
  Pubblicazione: (2024)
• DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs
  di: Lv, Minxuan, et al.
  Pubblicazione: (2025)