Scaling Laws for Upcycling Mixture-of-Experts Language Models

Documenti analoghi

• Reusing Overtrained Language Models Saturates Scaling
  di: Liew, Seng Pei, et al.
  Pubblicazione: (2025)
• Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints
  di: Liew, Seng Pei, et al.
  Pubblicazione: (2026)
• Upcycling Large Language Models into Mixture of Experts
  di: He, Ethan, et al.
  Pubblicazione: (2024)
• Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
  di: Nakamura, Taishi, et al.
  Pubblicazione: (2025)
• MoIN: Mixture of Introvert Experts to Upcycle an LLM
  di: Tejankar, Ajinkya, et al.
  Pubblicazione: (2024)