Cover Image

EMO: Pretraining Mixture of Experts for Emergent Modularity

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Wang, Ryan, Bhagia, Akshita, Min, Sewon
Format:	Preprint
Published:	2026
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Train Separately, Merge Together: Modular Post-Training with Mixture-of-Experts
by: Morrison, Jacob, et al.
Published: (2026)

OLMoE: Open Mixture-of-Experts Language Models
by: Muennighoff, Niklas, et al.
Published: (2024)

DataDecide: How to Predict Best Pretraining Data with Small Experiments
by: Magnusson, Ian, et al.
Published: (2025)

Mixture of Neuron Experts
by: Cheng, Runxi, et al.
Published: (2025)

MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Experts
by: Zhao, Yushu, et al.
Published: (2025)

EMO: Earth Mover Distance Optimization for Auto-Regressive Language Modeling
by: Ren, Siyu, et al.
Published: (2023)

Mixture of Lookup Experts
by: Jie, Shibo, et al.
Published: (2025)

Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization
by: Wang, Yaoxiang, et al.
Published: (2025)

Layerwise Recurrent Router for Mixture-of-Experts
by: Qiu, Zihan, et al.
Published: (2024)

BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models
by: Cao, Qingqing, et al.
Published: (2023)

nEMO: Dataset of Emotional Speech in Polish
by: Christop, Iwona
Published: (2024)

Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models
by: Guo, Hongcheng, et al.
Published: (2025)

Understanding the Mixture-of-Experts with Nadaraya-Watson Kernel
by: Zheng, Chuanyang, et al.
Published: (2025)

EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Models
by: Jing, Linglin, et al.
Published: (2025)

Router Upcycling: Leveraging Mixture-of-Routers in Mixture-of-Experts Upcycling
by: Ran, Junfeng, et al.
Published: (2025)

ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Experts
by: Tan, Zheyue, et al.
Published: (2025)

Unveiling Super Experts in Mixture-of-Experts Large Language Models
by: Su, Zunhai, et al.
Published: (2025)

PMoE: Progressive Mixture of Experts with Asymmetric Transformer for Continual Learning
by: Jung, Min Jae, et al.
Published: (2024)

Mixture of Experts for Low-Resource LLMs
by: Joseph, Ori Bar, et al.
Published: (2026)

Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steering
by: Chen, Yuxin, et al.
Published: (2026)

MEMoE: Enhancing Model Editing with Mixture of Experts Adaptors
by: Wang, Renzhi, et al.
Published: (2024)

Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMs
by: Bai, Jun, et al.
Published: (2025)

What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data
by: Movva, Rajiv, et al.
Published: (2025)

Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
by: Wang, Zihan, et al.
Published: (2025)

Maximum Score Routing For Mixture-of-Experts
by: Dong, Bowen, et al.
Published: (2025)

Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Experts
by: Jawahar, Ganesh, et al.
Published: (2023)

Multi-Head Mixture-of-Experts
by: Wu, Xun, et al.
Published: (2024)

Mixture Compressor for Mixture-of-Experts LLMs Gains More
by: Huang, Wei, et al.
Published: (2024)

Upcycling Large Language Models into Mixture of Experts
by: He, Ethan, et al.
Published: (2024)

MoDEM: Mixture of Domain Expert Models
by: Simonds, Toby, et al.
Published: (2024)

DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning
by: Bai, Sikai, et al.
Published: (2025)

EMO-KNOW: A Large Scale Dataset on Emotion and Emotion-cause
by: Nguyen, Mia Huong, et al.
Published: (2024)

EMO-Reasoning: Benchmarking Emotional Reasoning Capabilities in Spoken Dialogue Systems
by: Liu, Jingwen, et al.
Published: (2025)

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
by: Yoon, Youngsik, et al.
Published: (2026)

MidPO: Dual Preference Optimization for Safety and Helpfulness in Large Language Models via a Mixture of Experts Framework
by: Qi, Yupeng, et al.
Published: (2025)

Mixture-of-Retrieval Experts for Reasoning-Guided Multimodal Knowledge Exploitation
by: Peng, Chunyi, et al.
Published: (2025)

Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignment
by: Liu, Zhili, et al.
Published: (2024)

ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
by: He, Xin, et al.
Published: (2024)

Scaling Laws for Mixture Pretraining Under Data Constraints
by: Sedova, Anastasiia, et al.
Published: (2026)

Optimizing Pretraining Data Mixtures with LLM-Estimated Utility
by: Held, William, et al.
Published: (2025)