DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914620691709952 |
|---|---|
| author | Feng, Jiarui Zeng, Hanqing Grover, Karish Qiu, Ruizhong Xia, Yinglong Zhang, Qiang Wang, Qifan Chen, Ren Fu, Dongqi Liu, Jiayi Zhao, Zhoukai Fan, Xiangjun Zhang, Benyu Chen, Yixin |
| author_facet | Feng, Jiarui Zeng, Hanqing Grover, Karish Qiu, Ruizhong Xia, Yinglong Zhang, Qiang Wang, Qifan Chen, Ren Fu, Dongqi Liu, Jiayi Zhao, Zhoukai Fan, Xiangjun Zhang, Benyu Chen, Yixin |
| contents | Mixture-of-Experts (MoE) models have become a leading approach for decoupling parameter count from computational cost in large language models, yet effectively scaling MoE performance remains a challenge. Prior work shows that fine-grained experts enlarge the space of expert combinations and improve flexibility, but they also impose substantial routing overhead, creating a new scalability bottleneck. In this paper, we explore a complementary axis for scaling -- how expert outputs are aggregated. We theoretically show that replacing the standard weighted-summation aggregation with structural aggregation expands the expert-combination space without altering the experts or router, and enables possible multi-step reasoning within a single MoE layer. To this end, we propose DAG-MoE, a sparse MoE framework that employs a lightweight module to automatically learn the optimal aggregation structure among the selected experts. Extensive experiments under standard language modeling settings show that DAG-MoE consistently improves performance in both pretraining and fine-tuning, surpassing traditional MoE baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_01062 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts Feng, Jiarui Zeng, Hanqing Grover, Karish Qiu, Ruizhong Xia, Yinglong Zhang, Qiang Wang, Qifan Chen, Ren Fu, Dongqi Liu, Jiayi Zhao, Zhoukai Fan, Xiangjun Zhang, Benyu Chen, Yixin Artificial Intelligence Mixture-of-Experts (MoE) models have become a leading approach for decoupling parameter count from computational cost in large language models, yet effectively scaling MoE performance remains a challenge. Prior work shows that fine-grained experts enlarge the space of expert combinations and improve flexibility, but they also impose substantial routing overhead, creating a new scalability bottleneck. In this paper, we explore a complementary axis for scaling -- how expert outputs are aggregated. We theoretically show that replacing the standard weighted-summation aggregation with structural aggregation expands the expert-combination space without altering the experts or router, and enables possible multi-step reasoning within a single MoE layer. To this end, we propose DAG-MoE, a sparse MoE framework that employs a lightweight module to automatically learn the optimal aggregation structure among the selected experts. Extensive experiments under standard language modeling settings show that DAG-MoE consistently improves performance in both pretraining and fine-tuning, surpassing traditional MoE baselines. |
| title | DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2606.01062 |