Copertina

Revisiting MoE and Dense Speed-Accuracy Comparisons for LLM Training

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Du, Xianzhi, Gunter, Tom, Kong, Xiang, Lee, Mark, Wang, Zirui, Zhang, Aonan, Du, Nan, Pang, Ruoming
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Machine Learning Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations
di: Jaiswal, Ajay, et al.
Pubblicazione: (2025)

Can External Validation Tools Improve Annotation Quality for LLM-as-a-Judge?
di: Findeis, Arduin, et al.
Pubblicazione: (2025)

MoE-Loco: Mixture of Experts for Multitask Locomotion
di: Huang, Runhan, et al.
Pubblicazione: (2025)

MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE
di: Huang, Zongle, et al.
Pubblicazione: (2025)

Large Language Model-guided Document Selection
di: Kong, Xiang, et al.
Pubblicazione: (2024)

MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design
di: Duanmu, Haojie, et al.
Pubblicazione: (2025)

Analytical FFN-to-MoE Restructuring via Activation Pattern Analysis
di: Pei, Zehua, et al.
Pubblicazione: (2025)

DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-training
di: Jin, Can, et al.
Pubblicazione: (2025)

AquilaMoE: Efficient Training for MoE Models with Scale-Up and Scale-Out Strategies
di: Zhang, Bo-Wen, et al.
Pubblicazione: (2024)

FFT-MoE: Efficient Federated Fine-Tuning for Foundation Models via Large-scale Sparse MoE under Heterogeneous Edge
di: Hu, Gang, et al.
Pubblicazione: (2025)

SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget
di: Kong, Rui, et al.
Pubblicazione: (2023)

Synthetic bootstrapped pretraining
di: Yang, Zitong, et al.
Pubblicazione: (2025)

STAMImputer: Spatio-Temporal Attention MoE for Traffic Data Imputation
di: Wang, Yiming, et al.
Pubblicazione: (2025)

MoE-PHDS: One MoE checkpoint for flexible runtime sparsity
di: Hannah, Lauren. A, et al.
Pubblicazione: (2025)

A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems
di: Wu, Qi, et al.
Pubblicazione: (2026)

Deconstructing Pre-training: Knowledge Attribution Analysis in MoE and Dense Models
di: Wang, Bo, et al.
Pubblicazione: (2026)

LLaDA-MoE: A Sparse MoE Diffusion Language Model
di: Zhu, Fengqi, et al.
Pubblicazione: (2025)

GW-MoE: Resolving Uncertainty in MoE Router with Global Workspace Theory
di: Wu, Haoze, et al.
Pubblicazione: (2024)

D$^{2}$MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving
di: Wang, Haodong, et al.
Pubblicazione: (2025)

MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraints
di: Yuan, Yichao, et al.
Pubblicazione: (2025)

Grouter: Decoupling Routing from Representation for Accelerated MoE Training
di: Xu, Yuqi, et al.
Pubblicazione: (2026)

Revisiting Recommendation Loss Functions through Contrastive Learning (Technical Report)
di: Li, Dong, et al.
Pubblicazione: (2023)

BitsMoE: Efficient Spectral Energy-Guided Bit Allocation for MoE LLM Quantization
di: Zhao, Jiayu, et al.
Pubblicazione: (2026)

Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
di: Jiang, Fan, et al.
Pubblicazione: (2026)

Enhancing Online Recruitment with Category-Aware MoE and LLM-based Data Augmentation
di: Chen, Minping, et al.
Pubblicazione: (2026)

LocMoE: A Low-Overhead MoE for Large Language Model Training
di: Li, Jing, et al.
Pubblicazione: (2024)

MoCHA: Advanced Vision-Language Reasoning with MoE Connector and Hierarchical Group Attention
di: Pang, Yuqi, et al.
Pubblicazione: (2025)

ProMoE: Fast MoE-based LLM Serving using Proactive Caching
di: Song, Xiaoniu, et al.
Pubblicazione: (2024)

ShardMemo: Masked MoE Routing for Sharded Agentic LLM Memory
di: Zhao, Yang, et al.
Pubblicazione: (2026)

Evaluating Expert Contributions in a MoE LLM for Quiz-Based Tasks
di: Chernov, Andrei
Pubblicazione: (2025)

Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers
di: Ma, Wenhan, et al.
Pubblicazione: (2025)

MoEless: Efficient MoE LLM Serving via Serverless Computing
di: Yu, Hanfei, et al.
Pubblicazione: (2026)

Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data
di: Li, Yunxin, et al.
Pubblicazione: (2025)

KBVQ-MoE: KLT-guided SVD with Bias-Corrected Vector Quantization for MoE Large Language Models
di: Xu, Zukang, et al.
Pubblicazione: (2026)

DOT-MoE: Differentiable Optimal Transport for MoEfication
di: Bamba, Udbhav, et al.
Pubblicazione: (2026)

VEQ: Modality-Adaptive Quantization for MoE Vision-Language Models
di: Qin, Guangshuo, et al.
Pubblicazione: (2026)

MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models
di: Ye, Xin, et al.
Pubblicazione: (2026)

Practical FP4 Training for Large-Scale MoE Models on Hopper GPUs
di: Zhang, Wuyue, et al.
Pubblicazione: (2026)

MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs
di: Cao, Shiyi, et al.
Pubblicazione: (2024)

GazeFormer-MoE: Context-Aware Gaze Estimation via CLIP and MoE Transformer
di: Zhao, Xinyuan, et al.
Pubblicazione: (2026)