Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.00099 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912445406117888 |
|---|---|
| author | Skliar, Andrii van Rozendaal, Ties Lepert, Romain Boinovski, Todor van Baalen, Mart Nagel, Markus Whatmough, Paul Bejnordi, Babak Ehteshami |
| author_facet | Skliar, Andrii van Rozendaal, Ties Lepert, Romain Boinovski, Todor van Baalen, Mart Nagel, Markus Whatmough, Paul Bejnordi, Babak Ehteshami |
| contents | Mixture of Experts (MoE) LLMs have recently gained attention for their ability to enhance performance by selectively engaging specialized subnetworks or "experts" for each input. However, deploying MoEs on memory-constrained devices remains challenging, particularly when generating tokens sequentially with a batch size of one, as opposed to typical high-throughput settings involving long sequences or large batches. In this work, we optimize MoE on memory-constrained devices where only a subset of expert weights fit in DRAM. We introduce a novel cache-aware routing strategy that leverages expert reuse during token generation to improve cache locality. We evaluate our approach on language modeling, MMLU, and GSM8K benchmarks and present on-device results demonstrating 2$\times$ speedups on mobile devices, offering a flexible, training-free solution to extend MoE's applicability across real-world applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_00099 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference Skliar, Andrii van Rozendaal, Ties Lepert, Romain Boinovski, Todor van Baalen, Mart Nagel, Markus Whatmough, Paul Bejnordi, Babak Ehteshami Machine Learning Artificial Intelligence Hardware Architecture Mixture of Experts (MoE) LLMs have recently gained attention for their ability to enhance performance by selectively engaging specialized subnetworks or "experts" for each input. However, deploying MoEs on memory-constrained devices remains challenging, particularly when generating tokens sequentially with a batch size of one, as opposed to typical high-throughput settings involving long sequences or large batches. In this work, we optimize MoE on memory-constrained devices where only a subset of expert weights fit in DRAM. We introduce a novel cache-aware routing strategy that leverages expert reuse during token generation to improve cache locality. We evaluate our approach on language modeling, MMLU, and GSM8K benchmarks and present on-device results demonstrating 2$\times$ speedups on mobile devices, offering a flexible, training-free solution to extend MoE's applicability across real-world applications. |
| title | Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference |
| topic | Machine Learning Artificial Intelligence Hardware Architecture |
| url | https://arxiv.org/abs/2412.00099 |