Cover Image

Group then Scale: Dynamic Mixture-of-Experts Multilingual Language Model

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Li, Chong, Deng, Yingzhuo, Zhang, Jiajun, Zong, Chengqing
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Language Imbalance Driven Rewarding for Multilingual Self-improving
by: Yang, Wen, et al.
Published: (2024)

Parallel Scaling Law: Unveiling Reasoning Generalization through A Cross-Linguistic Perspective
by: Yang, Wen, et al.
Published: (2025)

X-Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions
by: Li, Chong, et al.
Published: (2024)

Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment
by: Yang, Wen, et al.
Published: (2025)

TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment
by: Li, Chong, et al.
Published: (2026)

Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment
by: Li, Chong, et al.
Published: (2023)

Routing-Aligned Fine-Tuning for Multilingual Downstream Tasks in Mixture-of-Experts Models
by: Deng, Guanzhi, et al.
Published: (2026)

ChineseWebText 2.0: Large-Scale High-quality Chinese Web Text with Multi-dimensional and fine-grained information
by: Zhang, Wanyue, et al.
Published: (2024)

Mixture of Heterogeneous Grouped Experts for Language Modeling
by: Ma, Zhicheng, et al.
Published: (2026)

Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
by: Jiang, Fan, et al.
Published: (2026)

Multilingual Routing in Mixture-of-Experts
by: Bandarkar, Lucas, et al.
Published: (2025)

Computational Lesions in Multilingual Language Models Separate Shared and Language-specific Brain Alignment
by: Cui, Yang, et al.
Published: (2026)

TokAlign: Efficient Vocabulary Adaptation via Token Alignment
by: Li, Chong, et al.
Published: (2025)

Computational Models to Study Language Processing in the Human Brain: A Survey
by: Wang, Shaonan, et al.
Published: (2024)

Component-Level Lesioning of Language Models Reveals Clinically Aligned Aphasia Phenotypes
by: Wang, Yifan, et al.
Published: (2026)

DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Models
by: Deng, Guanzhi, et al.
Published: (2026)

MapGuide: A Simple yet Effective Method to Reconstruct Continuous Language from Brain Activities
by: Zhao, Xinpei, et al.
Published: (2024)

Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
by: Liu, Boan, et al.
Published: (2023)

Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Models
by: Zhu, Shien, et al.
Published: (2025)

Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
by: Lu, Xudong, et al.
Published: (2024)

MultiPL-MoE: Multi-Programming-Lingual Extension of Large Language Models through Hybrid Mixture-of-Experts
by: Wang, Qing, et al.
Published: (2025)

SciDFM: A Large Language Model with Mixture-of-Experts for Science
by: Sun, Liangtai, et al.
Published: (2024)

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
by: DeepSeek-AI, et al.
Published: (2024)

Upcycling Large Language Models into Mixture of Experts
by: He, Ethan, et al.
Published: (2024)

Revisiting Multilingual Data Mixtures in Language Model Pretraining
by: Foroutan, Negar, et al.
Published: (2025)

Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models
by: Wei, Tianwen, et al.
Published: (2024)

Towards a Comprehensive Scaling Law of Mixture-of-Experts
by: Zhao, Guoliang, et al.
Published: (2025)

Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation
by: Liang, Yupu, et al.
Published: (2025)

Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignment
by: Liu, Zhili, et al.
Published: (2024)

Cosine-Similarity Routing with Semantic Anchors for Interpretable Mixture-of-Experts Language Models
by: Ternovtsii, Ivan, et al.
Published: (2025)

Flexible and Effective Mixing of Large Language Models into a Mixture of Domain Experts
by: Lee, Rhui Dih, et al.
Published: (2024)

Scaling Embeddings Outperforms Scaling Experts in Language Models
by: Liu, Hong, et al.
Published: (2026)

Aligning Multilingual Reasoning with Verifiable Semantics from a High-Resource Expert Model
by: Faisal, Fahim, et al.
Published: (2025)

Scaling Laws for Fine-Grained Mixture of Experts
by: Krajewski, Jakub, et al.
Published: (2024)

Pruning and Distilling Mixture-of-Experts into Dense Language Models
by: Kim, Junhyuck, et al.
Published: (2026)

OLMoE: Open Mixture-of-Experts Language Models
by: Muennighoff, Niklas, et al.
Published: (2024)

SparseDoctor: Towards Efficient Chat Doctor with Mixture of Experts Enhanced Large Language Models
by: Zhang, Jianbin, et al.
Published: (2025)

Yuan 2.0-M32: Mixture of Experts with Attention Router
by: Wu, Shaohua, et al.
Published: (2024)

GigaChat Family: Efficient Russian Language Modeling Through Mixture of Experts Architecture
by: GigaChat team, et al.
Published: (2025)

Routing by Analogy: kNN-Augmented Expert Assignment for Mixture-of-Experts
by: Lyu, Boxuan, et al.
Published: (2026)