Mozart: Modularized and Efficient MoE Training on 3.5D Wafer-Scale Chiplet Architectures
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918378060382208 |
|---|---|
| author | Luo, Shuqing Han, Ye Li, Pingzhi Qin, Jiayin Peng, Jie Yang Zhao Yu Cao Chen, Tianlong |
| author_facet | Luo, Shuqing Han, Ye Li, Pingzhi Qin, Jiayin Peng, Jie Yang Zhao Yu Cao Chen, Tianlong |
| contents | Mixture-of-Experts (MoE) architecture offers enhanced efficiency for Large Language Models (LLMs) with modularized computation, yet its inherent sparsity poses significant hardware deployment challenges, including memory locality issues, communication overhead, and inefficient computing resource utilization. Inspired by the modular organization of the human brain, we propose Mozart, a novel algorithm-hardware co-design framework tailored for efficient training of MoE-based LLMs on 3.5D wafer-scale chiplet architectures. On the algorithm side, Mozart exploits the inherent modularity of chiplets and introduces: (1) an expert allocation strategy that enables efficient on-package all-to-all communication, and (2) a fine-grained scheduling mechanism that improves communication-computation overlap through streaming tokens and experts. On the architecture side, Mozart adaptively co-locates heterogeneous modules on specialized chiplets with a 2.5D NoP-Tree topology and hierarchical memory structure. Evaluation across three popular MoE models demonstrates significant efficiency gains, enabling more effective parallelization and resource utilization for large-scale modularized MoE-LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_07006 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Mozart: Modularized and Efficient MoE Training on 3.5D Wafer-Scale Chiplet Architectures Luo, Shuqing Han, Ye Li, Pingzhi Qin, Jiayin Peng, Jie Yang Zhao Yu Cao Chen, Tianlong Hardware Architecture Mixture-of-Experts (MoE) architecture offers enhanced efficiency for Large Language Models (LLMs) with modularized computation, yet its inherent sparsity poses significant hardware deployment challenges, including memory locality issues, communication overhead, and inefficient computing resource utilization. Inspired by the modular organization of the human brain, we propose Mozart, a novel algorithm-hardware co-design framework tailored for efficient training of MoE-based LLMs on 3.5D wafer-scale chiplet architectures. On the algorithm side, Mozart exploits the inherent modularity of chiplets and introduces: (1) an expert allocation strategy that enables efficient on-package all-to-all communication, and (2) a fine-grained scheduling mechanism that improves communication-computation overlap through streaming tokens and experts. On the architecture side, Mozart adaptively co-locates heterogeneous modules on specialized chiplets with a 2.5D NoP-Tree topology and hierarchical memory structure. Evaluation across three popular MoE models demonstrates significant efficiency gains, enabling more effective parallelization and resource utilization for large-scale modularized MoE-LLMs. |
| title | Mozart: Modularized and Efficient MoE Training on 3.5D Wafer-Scale Chiplet Architectures |
| topic | Hardware Architecture |
| url | https://arxiv.org/abs/2603.07006 |