Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.17073 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917156672765952 |
|---|---|
| author | Liu, Zhenyu Liu, Yunzhen Fan, Zehao Gagnon, Garrett Hou, Yayue Wu, Nan Kang, Yangwook Liu, Liu |
| author_facet | Liu, Zhenyu Liu, Yunzhen Fan, Zehao Gagnon, Garrett Hou, Yayue Wu, Nan Kang, Yangwook Liu, Liu |
| contents | Mixture-of-Experts (MoE) models scale capacity via sparse activation but stress memory and bandwidth. Offloading alleviates GPU memory by fetching experts on demand, yet token-level routing causes irregular transfers that make inference I/O-bound. Static uniform quantization reduces traffic but degrades accuracy under aggressive compression by ignoring expert heterogeneity. We present Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensation, which performs router-guided precision restoration using precomputed low-rank compensators. At inference time, our method transfers compact low-rank factors with Top-n (n<k) experts per token and applies compensation to them, keeping others low-bit. Integrated with offloading on GPU and GPU-NDP systems, our method delivers a superior bandwidth-accuracy trade-off and improved throughput. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_17073 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensation Liu, Zhenyu Liu, Yunzhen Fan, Zehao Gagnon, Garrett Hou, Yayue Wu, Nan Kang, Yangwook Liu, Liu Machine Learning Mixture-of-Experts (MoE) models scale capacity via sparse activation but stress memory and bandwidth. Offloading alleviates GPU memory by fetching experts on demand, yet token-level routing causes irregular transfers that make inference I/O-bound. Static uniform quantization reduces traffic but degrades accuracy under aggressive compression by ignoring expert heterogeneity. We present Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensation, which performs router-guided precision restoration using precomputed low-rank compensators. At inference time, our method transfers compact low-rank factors with Top-n (n<k) experts per token and applies compensation to them, keeping others low-bit. Integrated with offloading on GPU and GPU-NDP systems, our method delivers a superior bandwidth-accuracy trade-off and improved throughput. |
| title | Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensation |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2512.17073 |