Copertina

Continual Pre-training of MoEs: How robust is your router?

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Thérien, Benjamin, Joseph, Charles-Étienne, Sarwar, Zain, Panda, Ashwinee, Das, Anirban, Zhang, Shi-Xiong, Rawls, Stephen, Sahu, Sambit, Belilovsky, Eugene, Rish, Irina
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Machine Learning Artificial Intelligence Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Dense Backpropagation Improves Training for Sparse Mixture-of-Experts
di: Panda, Ashwinee, et al.
Pubblicazione: (2025)

Beyond Cosine Decay: On the effectiveness of Infinite Learning Rate Schedule for Continual Pre-training
di: Singh, Vaibhav, et al.
Pubblicazione: (2025)

Warming Up for Zeroth-Order Federated Pre-Training with Low Resource Clients
di: Legate, Gwen, et al.
Pubblicazione: (2025)

MuLoCo: Muon is a practical inner optimizer for DiLoCo
di: Thérien, Benjamin, et al.
Pubblicazione: (2025)

$μ$LO: Compute-Efficient Meta-Generalization of Learned Optimizers
di: Thérien, Benjamin, et al.
Pubblicazione: (2024)

Simple and Scalable Strategies to Continually Pre-train Large Language Models
di: Ibrahim, Adam, et al.
Pubblicazione: (2024)

Communication Efficient LLM Pre-training with SparseLoCo
di: Sarfi, Amir, et al.
Pubblicazione: (2025)

LLM Surgery: Efficient Knowledge Unlearning and Editing in Large Language Models
di: Veldanda, Akshaj Kumar, et al.
Pubblicazione: (2024)

Meta-learning Optimizers for Communication-Efficient Learning
di: Joseph, Charles-Étienne, et al.
Pubblicazione: (2023)

Critique-Guided Distillation for Robust Reasoning via Refinement
di: Kapusuzoglu, Berkcan, et al.
Pubblicazione: (2025)

SPEAR-MM: Selective Parameter Evaluation and Restoration via Model Merging for Efficient Financial LLM Adaptation
di: Kapusuzoglu, Berkcan, et al.
Pubblicazione: (2025)

Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs
di: Li, Bo, et al.
Pubblicazione: (2026)

PyLO: Towards Accessible Learned Optimizers in PyTorch
di: Janson, Paul, et al.
Pubblicazione: (2025)

Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learning
di: Mircea, Andrei, et al.
Pubblicazione: (2025)

Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet
di: Lidin, Joel, et al.
Pubblicazione: (2026)

Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models
di: Abbes, Istabrak, et al.
Pubblicazione: (2025)

DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-training
di: Jin, Can, et al.
Pubblicazione: (2025)

Deconstructing Pre-training: Knowledge Attribution Analysis in MoE and Dense Models
di: Wang, Bo, et al.
Pubblicazione: (2026)

WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Average
di: Fournier, Louis, et al.
Pubblicazione: (2024)

Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL
di: Miahi, Erfan, et al.
Pubblicazione: (2026)

FairDropout: Using Example-Tied Dropout to Enhance Generalization of Minority Groups
di: Nanfack, Geraldin, et al.
Pubblicazione: (2025)

MPipeMoE: Memory Efficient MoE for Pre-trained Models with Adaptive Pipeline Parallelism
di: Zhang, Zheng, et al.
Pubblicazione: (2025)

Sparse Crosscoders for diffing MoEs and Dense models
di: Chaudhari, Marmik, et al.
Pubblicazione: (2026)

SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
di: Tang, Shengkun, et al.
Pubblicazione: (2026)

From Curated Data to Scalable Models: Continual Pre-training of Dense and MoE Large Language Models for Tibetan
di: Yang, Lei, et al.
Pubblicazione: (2025)

Heterogeneous Low-Bandwidth Pre-Training of LLMs
di: Obeidi, Yazan, et al.
Pubblicazione: (2026)

Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs
di: Ye, Charles, et al.
Pubblicazione: (2026)

Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
di: Wang, Qi, et al.
Pubblicazione: (2025)

EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE
di: Chen, Junyi, et al.
Pubblicazione: (2023)

RainbowPO: A Unified Framework for Combining Improvements in Preference Optimization
di: Zhao, Hanyang, et al.
Pubblicazione: (2024)

Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey
di: Winata, Genta Indra, et al.
Pubblicazione: (2024)

Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks
di: Davari, MohammadReza, et al.
Pubblicazione: (2023)

LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
di: Zhu, Tong, et al.
Pubblicazione: (2024)

A Guide to Robust Generalization: The Impact of Architecture, Pre-training, and Optimization Strategy
di: Heuillet, Maxime, et al.
Pubblicazione: (2025)

Dirichlet-Prior Shaping: Guiding Expert Specialization in Upcycled MoEs
di: Mirvakhabova, Leyla, et al.
Pubblicazione: (2025)

MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs
di: Zhou, Yuhang, et al.
Pubblicazione: (2025)

Insights into Martian bedform migration: Results from Gale, Jezero and Pasteur craters
di: Anurag Sahu, et al.
Pubblicazione: (2024)

Does a Global Perspective Help Prune Sparse MoEs Elegantly?
di: Zhang, Zeliang, et al.
Pubblicazione: (2026)

Stabilizing Native Low-Rank LLM Pretraining
di: Janson, Paul, et al.
Pubblicazione: (2026)

From Feature Visualization to Visual Circuits: Effect of Adversarial Model Manipulation
di: Nanfack, Geraldin, et al.
Pubblicazione: (2024)