WDMoE: Wireless Distributed Mixture of Experts for Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917833399599104 |
|---|---|
| author | Xue, Nan Sun, Yaping Chen, Zhiyong Tao, Meixia Xu, Xiaodong Qian, Liang Cui, Shuguang Zhang, Wenjun Zhang, Ping |
| author_facet | Xue, Nan Sun, Yaping Chen, Zhiyong Tao, Meixia Xu, Xiaodong Qian, Liang Cui, Shuguang Zhang, Wenjun Zhang, Ping |
| contents | Large Language Models (LLMs) have achieved significant success in various natural language processing tasks, but the role of wireless networks in supporting LLMs has not been thoroughly explored. In this paper, we propose a wireless distributed Mixture of Experts (WDMoE) architecture to enable collaborative deployment of LLMs across edge servers at the base station (BS) and mobile devices in wireless networks. Specifically, we decompose the MoE layer in LLMs by placing the gating network and the preceding neural network layer at BS, while distributing the expert networks among the devices. This deployment leverages the parallel inference capabilities of expert networks on mobile devices, effectively utilizing the limited computing and caching resources of these devices. Accordingly, we develop a performance metric for WDMoE-based LLMs, which accounts for both model capability and latency. To minimize the latency while maintaining accuracy, we jointly optimize expert selection and bandwidth allocation based on the performance metric. Moreover, we build a hardware testbed using NVIDIA Jetson kits to validate the effectiveness of WDMoE. Both theoretical simulations and practical hardware experiments demonstrate that the proposed method can significantly reduce the latency without compromising LLM performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_06681 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | WDMoE: Wireless Distributed Mixture of Experts for Large Language Models Xue, Nan Sun, Yaping Chen, Zhiyong Tao, Meixia Xu, Xiaodong Qian, Liang Cui, Shuguang Zhang, Wenjun Zhang, Ping Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing Information Theory Large Language Models (LLMs) have achieved significant success in various natural language processing tasks, but the role of wireless networks in supporting LLMs has not been thoroughly explored. In this paper, we propose a wireless distributed Mixture of Experts (WDMoE) architecture to enable collaborative deployment of LLMs across edge servers at the base station (BS) and mobile devices in wireless networks. Specifically, we decompose the MoE layer in LLMs by placing the gating network and the preceding neural network layer at BS, while distributing the expert networks among the devices. This deployment leverages the parallel inference capabilities of expert networks on mobile devices, effectively utilizing the limited computing and caching resources of these devices. Accordingly, we develop a performance metric for WDMoE-based LLMs, which accounts for both model capability and latency. To minimize the latency while maintaining accuracy, we jointly optimize expert selection and bandwidth allocation based on the performance metric. Moreover, we build a hardware testbed using NVIDIA Jetson kits to validate the effectiveness of WDMoE. Both theoretical simulations and practical hardware experiments demonstrate that the proposed method can significantly reduce the latency without compromising LLM performance. |
| title | WDMoE: Wireless Distributed Mixture of Experts for Large Language Models |
| topic | Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing Information Theory |
| url | https://arxiv.org/abs/2411.06681 |