Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.07890 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915705242255360 |
|---|---|
| author | Lv, Bo Tang, Chen Zheng, Zifan Yang, Bohao Zhao, Kun Liao, Ning Wang, Xiaoxing Xiong, Feiyu Li, Zhiyu Liu, Nayu Jiang, Jingchi |
| author_facet | Lv, Bo Tang, Chen Zheng, Zifan Yang, Bohao Zhao, Kun Liao, Ning Wang, Xiaoxing Xiong, Feiyu Li, Zhiyu Liu, Nayu Jiang, Jingchi |
| contents | Traditional Mixture-of-Experts (MoE) networks benefit from utilizing multiple smaller expert models as opposed to a single large network. However, these experts typically operate independently, leaving a question open about whether interconnecting these models could enhance the performance of MoE networks. In response, we introduce GRAPHMOE, a novel method aimed at augmenting the cognitive depth of language models via a self-rethinking mechanism constructed on Pseudo GraphMoE networks. GRAPHMOE employs a recurrent routing strategy to simulate iterative thinking steps, thereby facilitating the flow of information among expert nodes. We implement the GRAPHMOE architecture using Low-Rank Adaptation techniques (LoRA) and conduct extensive experiments on various benchmark datasets. The experimental results reveal that GRAPHMOE outperforms other LoRA based models, achieving state-of-the-art (SOTA) performance. Additionally, this study explores a novel recurrent routing strategy that may inspire further advancements in enhancing the reasoning capabilities of language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_07890 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanism Lv, Bo Tang, Chen Zheng, Zifan Yang, Bohao Zhao, Kun Liao, Ning Wang, Xiaoxing Xiong, Feiyu Li, Zhiyu Liu, Nayu Jiang, Jingchi Computation and Language Artificial Intelligence Traditional Mixture-of-Experts (MoE) networks benefit from utilizing multiple smaller expert models as opposed to a single large network. However, these experts typically operate independently, leaving a question open about whether interconnecting these models could enhance the performance of MoE networks. In response, we introduce GRAPHMOE, a novel method aimed at augmenting the cognitive depth of language models via a self-rethinking mechanism constructed on Pseudo GraphMoE networks. GRAPHMOE employs a recurrent routing strategy to simulate iterative thinking steps, thereby facilitating the flow of information among expert nodes. We implement the GRAPHMOE architecture using Low-Rank Adaptation techniques (LoRA) and conduct extensive experiments on various benchmark datasets. The experimental results reveal that GRAPHMOE outperforms other LoRA based models, achieving state-of-the-art (SOTA) performance. Additionally, this study explores a novel recurrent routing strategy that may inspire further advancements in enhancing the reasoning capabilities of language models. |
| title | GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanism |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2501.07890 |