Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913584064233472 |
|---|---|
| author | Chen, Zeren Wang, Ziqin Wang, Zhen Liu, Huayang Yin, Zhenfei Liu, Si Sheng, Lu Ouyang, Wanli Qiao, Yu Shao, Jing |
| author_facet | Chen, Zeren Wang, Ziqin Wang, Zhen Liu, Huayang Yin, Zhenfei Liu, Si Sheng, Lu Ouyang, Wanli Qiao, Yu Shao, Jing |
| contents | Recent studies have demonstrated Large Language Models (LLMs) can extend their zero-shot generalization capabilities to multimodal learning through instruction tuning. As more modalities and downstream tasks are introduced, negative conflicts and interference may have a worse impact on performance. While this phenomenon has been overlooked in previous work, we propose a novel and extensible framework, called Octavius, for comprehensive studies and experimentation on multimodal learning with Multimodal Large Language Models (MLLMs). Specifically, we combine the well-known Mixture-of-Experts (MoE) and one of the representative PEFT techniques, i.e., LoRA, designing a novel LLM-based decoder, called LoRA-MoE, for multimodal learning. To the best of our knowledge, we are one of the pioneering efforts to introduce MoE into MLLMs to address this problem. The experimental results (about 20% improvement) have shown the effectiveness and versatility of our design in various 2D and 3D downstream tasks. Code and datasets are available at https://openlamm.github.io/tutorial/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_02684 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE Chen, Zeren Wang, Ziqin Wang, Zhen Liu, Huayang Yin, Zhenfei Liu, Si Sheng, Lu Ouyang, Wanli Qiao, Yu Shao, Jing Computer Vision and Pattern Recognition Computation and Language Recent studies have demonstrated Large Language Models (LLMs) can extend their zero-shot generalization capabilities to multimodal learning through instruction tuning. As more modalities and downstream tasks are introduced, negative conflicts and interference may have a worse impact on performance. While this phenomenon has been overlooked in previous work, we propose a novel and extensible framework, called Octavius, for comprehensive studies and experimentation on multimodal learning with Multimodal Large Language Models (MLLMs). Specifically, we combine the well-known Mixture-of-Experts (MoE) and one of the representative PEFT techniques, i.e., LoRA, designing a novel LLM-based decoder, called LoRA-MoE, for multimodal learning. To the best of our knowledge, we are one of the pioneering efforts to introduce MoE into MLLMs to address this problem. The experimental results (about 20% improvement) have shown the effectiveness and versatility of our design in various 2D and 3D downstream tasks. Code and datasets are available at https://openlamm.github.io/tutorial/. |
| title | Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2311.02684 |