Image de couverture de livre

Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Belenki, Lior, Agarwal, Alekh, Shi, Tianze, Toutanova, Kristina
Format:	Preprint
Publié:	2025
Sujets:	Machine Learning Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models
par: Zheng, Chen, et autres
Publié: (2025)

Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training
par: Zhong, Zexuan, et autres
Publié: (2024)

Mixture of Lookup Experts
par: Jie, Shibo, et autres
Publié: (2025)

Mixture Compressor for Mixture-of-Experts LLMs Gains More
par: Huang, Wei, et autres
Publié: (2024)

Scalable Training of Mixture-of-Experts Models with Megatron Core
par: Yan, Zijie, et autres
Publié: (2026)

Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
par: Wang, Zihan, et autres
Publié: (2025)

Bayesian Mixture of Experts For Large Language Models
par: Dialameh, Maryam, et autres
Publié: (2025)

Mixture of Heterogeneous Grouped Experts for Language Modeling
par: Ma, Zhicheng, et autres
Publié: (2026)

Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?
par: Hayase, Jonathan, et autres
Publié: (2024)

Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
par: Pan, Bowen, et autres
Publié: (2024)

Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models
par: Qiu, Zihan, et autres
Publié: (2025)

Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance
par: Ye, Jiasheng, et autres
Publié: (2024)

Scaling Laws for Upcycling Mixture-of-Experts Language Models
par: Liew, Seng Pei, et autres
Publié: (2025)

A Survey on Mixture of Experts in Large Language Models
par: Cai, Weilin, et autres
Publié: (2024)

Geometry-Preserving Aggregation for Mixture-of-Experts Embedding Models
par: Kachuee, Sajjad, et autres
Publié: (2026)

HMoE: Heterogeneous Mixture of Experts for Language Modeling
par: Wang, An, et autres
Publié: (2024)

Mixture of Experts Made Intrinsically Interpretable
par: Yang, Xingyi, et autres
Publié: (2025)

Maximum Score Routing For Mixture-of-Experts
par: Dong, Bowen, et autres
Publié: (2025)

Mixture of LoRA Experts
par: Wu, Xun, et autres
Publié: (2024)

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
par: Yoon, Youngsik, et autres
Publié: (2026)

Nexus: Specialization meets Adaptability for Efficiently Training Mixture of Experts
par: Gritsch, Nikolas, et autres
Publié: (2024)

Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss
par: Lv, Ang, et autres
Publié: (2025)

$\infty$-MoE: Generalizing Mixture of Experts to Infinite Experts
par: Takashiro, Shota, et autres
Publié: (2026)

Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning
par: He, Haoze, et autres
Publié: (2026)

A Closer Look into Mixture-of-Experts in Large Language Models
par: Lo, Ka Man, et autres
Publié: (2024)

Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free
par: Li, Ziyue, et autres
Publié: (2024)

Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixtures
par: Nguyen, Dang, et autres
Publié: (2024)

MoFE: Mixture of Frozen Experts Architecture
par: Seo, Jean, et autres
Publié: (2025)

Slicing and Dicing: Configuring Optimal Mixtures of Experts
par: Li, Margaret, et autres
Publié: (2026)

Binary-Integer-Programming Based Algorithm for Expert Load Balancing in Mixture-of-Experts Models
par: Sun, Yuan
Publié: (2025)

Upcycling Large Language Models into Mixture of Experts
par: He, Ethan, et autres
Publié: (2024)

Multilingual Routing in Mixture-of-Experts
par: Bandarkar, Lucas, et autres
Publié: (2025)

Routing-Free Mixture-of-Experts
par: Liu, Yilun, et autres
Publié: (2026)

Multi-Head Mixture-of-Experts
par: Wu, Xun, et autres
Publié: (2024)

OLMoE: Open Mixture-of-Experts Language Models
par: Muennighoff, Niklas, et autres
Publié: (2024)

Bridging Kolmogorov Complexity and Deep Learning: Asymptotically Optimal Description Length Objectives for Transformers
par: Shaw, Peter, et autres
Publié: (2025)

Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
par: Zhang, Zeliang, et autres
Publié: (2024)

Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts
par: Ahrac, Sagi, et autres
Publié: (2026)

Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
par: Nakamura, Taishi, et autres
Publié: (2025)

MoLAE: Mixture of Latent Experts for Parameter-Efficient Language Models
par: Liu, Zehua, et autres
Publié: (2025)