Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.09281 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914548780367872 |
|---|---|
| author | Gu, Hongyaoxing Chen, Xinzhe Hu, Lijuan Liu, Fangfang |
| author_facet | Gu, Hongyaoxing Chen, Xinzhe Hu, Lijuan Liu, Fangfang |
| contents | Mixture-of-Experts (MoE) models achieve remarkable performance by sparsely activating specialized experts, yet their massive parameters in experts pose significant challenges for deployment. While low-rank quantization offers a promising route to compress MoE models, existing methods still incur nonnegligible memory overhead and inference latency. To address these limitations, we propose \textsc{TileQ}, a fine-tuning-free post-training quantization (PTQ) method that employs 2D-tiling structured low-rank quantization to share low-rank factors across both input and output dimensions of MoE experts. Furthermore, we introduce an efficient inference technique for \textsc{TileQ} that fuses multiple low-rank expert computations into a single-pass operation, significantly improving hardware utilization. Experiments show that \textsc{TileQ} cuts down additional memory usage up to 10$\times$ and reduces inference latency to $\sim$5\% while preserving state-of-the-art accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_09281 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | TileQ: Efficient Low-Rank Quantization of Mixture-of-Experts with 2D Tiling Gu, Hongyaoxing Chen, Xinzhe Hu, Lijuan Liu, Fangfang Machine Learning Mixture-of-Experts (MoE) models achieve remarkable performance by sparsely activating specialized experts, yet their massive parameters in experts pose significant challenges for deployment. While low-rank quantization offers a promising route to compress MoE models, existing methods still incur nonnegligible memory overhead and inference latency. To address these limitations, we propose \textsc{TileQ}, a fine-tuning-free post-training quantization (PTQ) method that employs 2D-tiling structured low-rank quantization to share low-rank factors across both input and output dimensions of MoE experts. Furthermore, we introduce an efficient inference technique for \textsc{TileQ} that fuses multiple low-rank expert computations into a single-pass operation, significantly improving hardware utilization. Experiments show that \textsc{TileQ} cuts down additional memory usage up to 10$\times$ and reduces inference latency to $\sim$5\% while preserving state-of-the-art accuracy. |
| title | TileQ: Efficient Low-Rank Quantization of Mixture-of-Experts with 2D Tiling |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2605.09281 |