Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912439157653504 |
|---|---|
| author | Zhao, Ziyu Zhou, Yixiao Zhang, Zhi Zhu, Didi Shen, Tao Li, Zexi Yang, Jinluan Wang, Xuwu Su, Jing Kuang, Kun Wei, Zhongyu Wu, Fei Cheng, Yu |
| author_facet | Zhao, Ziyu Zhou, Yixiao Zhang, Zhi Zhu, Didi Shen, Tao Li, Zexi Yang, Jinluan Wang, Xuwu Su, Jing Kuang, Kun Wei, Zhongyu Wu, Fei Cheng, Yu |
| contents | Low-Rank Adaptation (LoRA) is widely used for adapting large language models (LLMs) to specific domains due to its efficiency and modularity. Meanwhile, vanilla LoRA struggles with task conflicts in multi-task scenarios. Recent works adopt Mixture of Experts (MoE) by treating each LoRA module as an expert, thereby mitigating task interference through multiple specialized LoRA modules. While effective, these methods often isolate knowledge within individual tasks, failing to fully exploit the shared knowledge across related tasks. In this paper, we establish a connection between single LoRA and multi-LoRA MoE, integrating them into a unified framework. We demonstrate that the dynamic routing of multiple LoRAs is functionally equivalent to rank partitioning and block-level activation within a single LoRA. We further empirically demonstrate that finer-grained LoRA partitioning, within the same total and activated parameter constraints, leads to better performance gains across heterogeneous tasks. Building on these findings, we propose Single-ranked Mixture of Experts LoRA (\textbf{SMoRA}), which embeds MoE into LoRA by \textit{treating each rank as an independent expert}. With a \textit{dynamic rank-wise activation} mechanism, SMoRA promotes finer-grained knowledge sharing while mitigating task conflicts. Experiments demonstrate that SMoRA activates fewer parameters yet achieves better performance in multi-task scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_15103 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning Zhao, Ziyu Zhou, Yixiao Zhang, Zhi Zhu, Didi Shen, Tao Li, Zexi Yang, Jinluan Wang, Xuwu Su, Jing Kuang, Kun Wei, Zhongyu Wu, Fei Cheng, Yu Machine Learning Artificial Intelligence Low-Rank Adaptation (LoRA) is widely used for adapting large language models (LLMs) to specific domains due to its efficiency and modularity. Meanwhile, vanilla LoRA struggles with task conflicts in multi-task scenarios. Recent works adopt Mixture of Experts (MoE) by treating each LoRA module as an expert, thereby mitigating task interference through multiple specialized LoRA modules. While effective, these methods often isolate knowledge within individual tasks, failing to fully exploit the shared knowledge across related tasks. In this paper, we establish a connection between single LoRA and multi-LoRA MoE, integrating them into a unified framework. We demonstrate that the dynamic routing of multiple LoRAs is functionally equivalent to rank partitioning and block-level activation within a single LoRA. We further empirically demonstrate that finer-grained LoRA partitioning, within the same total and activated parameter constraints, leads to better performance gains across heterogeneous tasks. Building on these findings, we propose Single-ranked Mixture of Experts LoRA (\textbf{SMoRA}), which embeds MoE into LoRA by \textit{treating each rank as an independent expert}. With a \textit{dynamic rank-wise activation} mechanism, SMoRA promotes finer-grained knowledge sharing while mitigating task conflicts. Experiments demonstrate that SMoRA activates fewer parameters yet achieves better performance in multi-task scenarios. |
| title | Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2501.15103 |