MoNDE: Mixture of Near-Data Experts for Large-Scale Sparse Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929363648249856 |
|---|---|
| author | Kim, Taehyun Choi, Kwanseok Cho, Youngmock Cho, Jaehoon Lee, Hyuk-Jae Sim, Jaewoong |
| author_facet | Kim, Taehyun Choi, Kwanseok Cho, Youngmock Cho, Jaehoon Lee, Hyuk-Jae Sim, Jaewoong |
| contents | Mixture-of-Experts (MoE) large language models (LLM) have memory requirements that often exceed the GPU memory capacity, requiring costly parameter movement from secondary memories to the GPU for expert computation. In this work, we present Mixture of Near-Data Experts (MoNDE), a near-data computing solution that efficiently enables MoE LLM inference. MoNDE reduces the volume of MoE parameter movement by transferring only the $\textit{hot}$ experts to the GPU, while computing the remaining $\textit{cold}$ experts inside the host memory device. By replacing the transfers of massive expert parameters with the ones of small activations, MoNDE enables far more communication-efficient MoE inference, thereby resulting in substantial speedups over the existing parameter offloading frameworks for both encoder and decoder operations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_18832 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MoNDE: Mixture of Near-Data Experts for Large-Scale Sparse Models Kim, Taehyun Choi, Kwanseok Cho, Youngmock Cho, Jaehoon Lee, Hyuk-Jae Sim, Jaewoong Machine Learning Artificial Intelligence Hardware Architecture Mixture-of-Experts (MoE) large language models (LLM) have memory requirements that often exceed the GPU memory capacity, requiring costly parameter movement from secondary memories to the GPU for expert computation. In this work, we present Mixture of Near-Data Experts (MoNDE), a near-data computing solution that efficiently enables MoE LLM inference. MoNDE reduces the volume of MoE parameter movement by transferring only the $\textit{hot}$ experts to the GPU, while computing the remaining $\textit{cold}$ experts inside the host memory device. By replacing the transfers of massive expert parameters with the ones of small activations, MoNDE enables far more communication-efficient MoE inference, thereby resulting in substantial speedups over the existing parameter offloading frameworks for both encoder and decoder operations. |
| title | MoNDE: Mixture of Near-Data Experts for Large-Scale Sparse Models |
| topic | Machine Learning Artificial Intelligence Hardware Architecture |
| url | https://arxiv.org/abs/2405.18832 |