DynaMo: Runtime Switchable Quantization for MoE with Cross-Dataset Adaptation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917191055572992 |
|---|---|
| author | Zheng, Zihao Cui, Xiuping Zheng, Size Li, Maoliang Chen, Jiayu Liang, Yun Chen, Xiang |
| author_facet | Zheng, Zihao Cui, Xiuping Zheng, Size Li, Maoliang Chen, Jiayu Liang, Yun Chen, Xiang |
| contents | As the Mix-of-Experts (MoE) architecture increases the number of parameters in large models, there is an even greater need for model quantization. However, existing quantization methods overlook the expert dynamics of MoE across multiple datasets. Moreover, the existing static quantization cannot adapt MoE to various data change scenarios. In this paper, we perform a multi-level analysis to reveal MoE dynamics and define the significance of each channel/each expert. Based on the analysis results, we propose \textit{DynaMo}, an end-to-end MoE quantization framework. DynaMo adopts an expert-level mixed-precision baseline quantization strategy, which ensures the quantized MoEs are compatible with multiple existing datasets. Furthermore, DynaMo incorporates a channel-level dynamic switching mechanism to adapt these quantized MoE models to novel datasets. Experiments show that DynaMo achieves a 2.78~4.54 PPL decrease and a 1.85%~3.77% accuracy improvement in various datasets, with ~3x inference speedup and negligible overhead. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_21135 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DynaMo: Runtime Switchable Quantization for MoE with Cross-Dataset Adaptation Zheng, Zihao Cui, Xiuping Zheng, Size Li, Maoliang Chen, Jiayu Liang, Yun Chen, Xiang Machine Learning As the Mix-of-Experts (MoE) architecture increases the number of parameters in large models, there is an even greater need for model quantization. However, existing quantization methods overlook the expert dynamics of MoE across multiple datasets. Moreover, the existing static quantization cannot adapt MoE to various data change scenarios. In this paper, we perform a multi-level analysis to reveal MoE dynamics and define the significance of each channel/each expert. Based on the analysis results, we propose \textit{DynaMo}, an end-to-end MoE quantization framework. DynaMo adopts an expert-level mixed-precision baseline quantization strategy, which ensures the quantized MoEs are compatible with multiple existing datasets. Furthermore, DynaMo incorporates a channel-level dynamic switching mechanism to adapt these quantized MoE models to novel datasets. Experiments show that DynaMo achieves a 2.78~4.54 PPL decrease and a 1.85%~3.77% accuracy improvement in various datasets, with ~3x inference speedup and negligible overhead. |
| title | DynaMo: Runtime Switchable Quantization for MoE with Cross-Dataset Adaptation |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2503.21135 |