Cover Image

Towards a Comprehensive Scaling Law of Mixture-of-Experts

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhao, Guoliang, Fu, Yuhan, Li, Shuaipeng, Sun, Xingwu, Xie, Ruobing, Wang, An, Han, Weidong, Yang, Zhen, Sun, Weixuan, Zhang, Yudong, Xu, Cheng-zhong, Wang, Di, Jiang, Jie
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

HMoE: Heterogeneous Mixture of Experts for Language Modeling
by: Wang, An, et al.
Published: (2024)

Self-Distillation for Multi-Token Prediction
by: Zhao, Guoliang, et al.
Published: (2026)

Scaling Laws for Floating Point Quantization Training
by: Sun, Xingwu, et al.
Published: (2025)

TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Model
by: Li, Yixing, et al.
Published: (2025)

PIP: Detecting Adversarial Examples in Large Vision-Language Models via Attention Patterns of Irrelevant Probe Questions
by: Zhang, Yudong, et al.
Published: (2024)

Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization
by: Fu, Yuhan, et al.
Published: (2024)

More Expressive Attention with Negative Weights
by: Lv, Ang, et al.
Published: (2024)

The Security Threat of Compressed Projectors in Large Vision-Language Models
by: Zhang, Yudong, et al.
Published: (2025)

Enhancing Contrastive Learning Inspired by the Philosophy of "The Blind Men and the Elephant"
by: Zhang, Yudong, et al.
Published: (2024)

QAVA: Query-Agnostic Visual Attack to Large Vision-Language Models
by: Zhang, Yudong, et al.
Published: (2025)

Autonomy-of-Experts Models
by: Lv, Ang, et al.
Published: (2025)

Fighting Fire with Fire (F3): A Training-free and Efficient Visual Adversarial Example Purification Method in LVLMs
by: Zhang, Yudong, et al.
Published: (2025)

DHCP: Detecting Hallucinations by Cross-modal Attention Pattern in Large Vision-Language Models
by: Zhang, Yudong, et al.
Published: (2024)

MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs
by: Ding, Wei, et al.
Published: (2026)

Proximal Supervised Fine-Tuning
by: Zhu, Wenhong, et al.
Published: (2025)

Magnifier Prompt: Tackling Multimodal Hallucination via Extremely Simple Instructions
by: Fu, Yuhan, et al.
Published: (2024)

Lossless KV Cache Compression to 2%
by: Yang, Zhen, et al.
Published: (2024)

Continuous Speech Tokenizer in Text To Speech
by: Li, Yixing, et al.
Published: (2024)

Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization
by: Wan, Weilin, et al.
Published: (2026)

PhD: A ChatGPT-Prompted Visual hallucination Evaluation Dataset
by: Liu, Jiazhen, et al.
Published: (2024)

Exploring Forgetting in Large Language Model Pre-Training
by: Liao, Chonghua, et al.
Published: (2024)

The Climb Carves Wisdom Deeper Than the Summit: On the Noisy Rewards in Learning to Reason
by: Lv, Ang, et al.
Published: (2025)

The Elephant in the Room: Rethinking the Usage of Pre-trained Language Model in Sequential Recommendation
by: Qu, Zekai, et al.
Published: (2024)

Language Models "Grok" to Copy
by: Lv, Ang, et al.
Published: (2024)

Scaling Laws for Fine-Grained Mixture of Experts
by: Krajewski, Jakub, et al.
Published: (2024)

Generalization and Scaling Laws for Mixture-of-Experts Transformers
by: Mayaki, Mansour Zoubeirou a
Published: (2026)

Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models
by: Tian, Changxin, et al.
Published: (2025)

Truth Forest: Toward Multi-Scale Truthfulness in Large Language Models through Intervention without Tuning
by: Chen, Zhongzhi, et al.
Published: (2023)

HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts
by: Zhao, Hao, et al.
Published: (2024)

Scaling Laws for Upcycling Mixture-of-Experts Language Models
by: Liew, Seng Pei, et al.
Published: (2025)

Multi-Grained Patch Training for Efficient LLM-based Recommendation
by: Liao, Jiayi, et al.
Published: (2025)

Layerwise Recurrent Router for Mixture-of-Experts
by: Qiu, Zihan, et al.
Published: (2024)

Mixture-of-Experts Operator Transformer for Large-Scale PDE Pre-Training
by: Wang, Hong, et al.
Published: (2025)

Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling
by: Li, Shuaipeng, et al.
Published: (2024)

Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization
by: Song, Guanghui, et al.
Published: (2025)

InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition
by: Liu, Fengze, et al.
Published: (2026)

Mixture of Lookup Experts
by: Jie, Shibo, et al.
Published: (2025)

Mixture of Diverse Size Experts
by: Sun, Manxi, et al.
Published: (2024)

RosePO: Aligning LLM-based Recommenders with Human Values
by: Liao, Jiayi, et al.
Published: (2024)

Scaling Machine Learning Interatomic Potentials with Mixtures of Experts
by: Liu, Yuzhi, et al.
Published: (2026)