RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917531116109824 |
|---|---|
| author | Sun, Mengyang Dou, Maochuan Feng, Tao Zhang, Dan Wang, Yihao Liu, Junpeng Zhu, Yifan Tang, Jie |
| author_facet | Sun, Mengyang Dou, Maochuan Feng, Tao Zhang, Dan Wang, Yihao Liu, Junpeng Zhu, Yifan Tang, Jie |
| contents | While Large Language Models (LLMs) are commonly fine-tuned to handle domain-specific tasks before being applied to vertical applications, adapting them to complex scenarios with diverse specialized knowledge remains challenging. Meanwhile, Mixture-of-Experts (MoE) architecture has risen as a crucial paradigm for training LLMs, and some recent works have also incorporated MoE into Parameter-Efficient Fine-Tuning (PEFT) to propose the Mixture of Low-rank Experts (MoE-LoRA), to enhance the power of low-rank adapters for learning complicated knowledge. However, conventional gating mechanisms in MoE typically apply only a scalar reweighing to selected experts, thereby limiting their underlying capacity of representation and generalization. Motivated and enabled by the low-rank structures in MoE-LoRA, we propose RotMoLE, a specialized MoE framework for low-rank experts featuring an additional rotation gate. Beyond simple scaling, RotMoLE implements a rotation mechanism for each selected expert, enabling superior expert exploitation and specialization for learning diverse data, especially when expert candidates are limited. Empirical results on complex multi-task and multilingual training scenarios validate our effectiveness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_25565 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism Sun, Mengyang Dou, Maochuan Feng, Tao Zhang, Dan Wang, Yihao Liu, Junpeng Zhu, Yifan Tang, Jie Machine Learning Computation and Language While Large Language Models (LLMs) are commonly fine-tuned to handle domain-specific tasks before being applied to vertical applications, adapting them to complex scenarios with diverse specialized knowledge remains challenging. Meanwhile, Mixture-of-Experts (MoE) architecture has risen as a crucial paradigm for training LLMs, and some recent works have also incorporated MoE into Parameter-Efficient Fine-Tuning (PEFT) to propose the Mixture of Low-rank Experts (MoE-LoRA), to enhance the power of low-rank adapters for learning complicated knowledge. However, conventional gating mechanisms in MoE typically apply only a scalar reweighing to selected experts, thereby limiting their underlying capacity of representation and generalization. Motivated and enabled by the low-rank structures in MoE-LoRA, we propose RotMoLE, a specialized MoE framework for low-rank experts featuring an additional rotation gate. Beyond simple scaling, RotMoLE implements a rotation mechanism for each selected expert, enabling superior expert exploitation and specialization for learning diverse data, especially when expert candidates are limited. Empirical results on complex multi-task and multilingual training scenarios validate our effectiveness. |
| title | RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2605.25565 |