MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910934826483712 |
|---|---|
| author | Duanmu, Haojie Li, Xiuhong Yuan, Zhihang Zheng, Size Duan, Jiangfei Zhang, Xingcheng Lin, Dahua |
| author_facet | Duanmu, Haojie Li, Xiuhong Yuan, Zhihang Zheng, Size Duan, Jiangfei Zhang, Xingcheng Lin, Dahua |
| contents | Mixture-of-Experts (MoE) models face deployment challenges due to their large parameter counts and computational demands. We explore quantization for MoE models and highlight two key insights: 1) linear blocks exhibit varying quantization sensitivity, and 2) divergent expert activation frequencies create heterogeneous computational characteristics. Based on these observations, we introduce MxMoE, a mixed-precision optimization framework for MoE models that considers both algorithmic and system perspectives. MxMoE navigates the design space defined by parameter sensitivity, expert activation dynamics, and hardware resources to derive efficient mixed-precision configurations. Additionally, MxMoE automatically generates optimized mixed-precision GroupGEMM kernels, enabling parallel execution of GEMMs with different precisions. Evaluations show that MxMoE outperforms existing methods, achieving 2.4 lower Wikitext-2 perplexity than GPTQ at 2.25-bit and delivering up to 3.4x speedup over full precision, as well as up to 29.4% speedup over uniform quantization at equivalent accuracy with 5-bit weight-activation quantization. Our code is available at https://github.com/cat538/MxMoE. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_05799 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design Duanmu, Haojie Li, Xiuhong Yuan, Zhihang Zheng, Size Duan, Jiangfei Zhang, Xingcheng Lin, Dahua Machine Learning Artificial Intelligence Mixture-of-Experts (MoE) models face deployment challenges due to their large parameter counts and computational demands. We explore quantization for MoE models and highlight two key insights: 1) linear blocks exhibit varying quantization sensitivity, and 2) divergent expert activation frequencies create heterogeneous computational characteristics. Based on these observations, we introduce MxMoE, a mixed-precision optimization framework for MoE models that considers both algorithmic and system perspectives. MxMoE navigates the design space defined by parameter sensitivity, expert activation dynamics, and hardware resources to derive efficient mixed-precision configurations. Additionally, MxMoE automatically generates optimized mixed-precision GroupGEMM kernels, enabling parallel execution of GEMMs with different precisions. Evaluations show that MxMoE outperforms existing methods, achieving 2.4 lower Wikitext-2 perplexity than GPTQ at 2.25-bit and delivering up to 3.4x speedup over full precision, as well as up to 29.4% speedup over uniform quantization at equivalent accuracy with 5-bit weight-activation quantization. Our code is available at https://github.com/cat538/MxMoE. |
| title | MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2505.05799 |