Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2504.08378 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911170476113920 |
|---|---|
| author | Jia, Fucheng Wu, Zewen Jiang, Shiqi Jiang, Huiqiang Zhang, Qianxi Yang, Yuqing Liu, Yunxin Ren, Ju Zhang, Deyu Cao, Ting |
| author_facet | Jia, Fucheng Wu, Zewen Jiang, Shiqi Jiang, Huiqiang Zhang, Qianxi Yang, Yuqing Liu, Yunxin Ren, Ju Zhang, Deyu Cao, Ting |
| contents | Large language models (LLMs) are increasingly being deployed on mobile devices, but the limited DRAM capacity constrains the deployable model size. This paper introduces ActiveFlow, the first LLM inference framework that can achieve adaptive DRAM usage for modern LLMs (not ReLU-based), enabling the scaling up of deployable model sizes. The framework is based on the novel concept of active weight DRAM-flash swapping and incorporates three novel techniques: (1) Cross-layer active weights preloading. It uses the activations from the current layer to predict the active weights of several subsequent layers, enabling computation and data loading to overlap, as well as facilitating large I/O transfers. (2) Sparsity-aware self-distillation. It adjusts the active weights to align with the dense-model output distribution, compensating for approximations introduced by contextual sparsity. (3) Active weight DRAM-flash swapping pipeline. It orchestrates the DRAM space allocation among the hot weight cache, preloaded active weights, and computation-involved weights based on available memory. Results show ActiveFlow achieves the performance-cost Pareto frontier compared to existing efficiency optimization methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_08378 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flash Jia, Fucheng Wu, Zewen Jiang, Shiqi Jiang, Huiqiang Zhang, Qianxi Yang, Yuqing Liu, Yunxin Ren, Ju Zhang, Deyu Cao, Ting Machine Learning Large language models (LLMs) are increasingly being deployed on mobile devices, but the limited DRAM capacity constrains the deployable model size. This paper introduces ActiveFlow, the first LLM inference framework that can achieve adaptive DRAM usage for modern LLMs (not ReLU-based), enabling the scaling up of deployable model sizes. The framework is based on the novel concept of active weight DRAM-flash swapping and incorporates three novel techniques: (1) Cross-layer active weights preloading. It uses the activations from the current layer to predict the active weights of several subsequent layers, enabling computation and data loading to overlap, as well as facilitating large I/O transfers. (2) Sparsity-aware self-distillation. It adjusts the active weights to align with the dense-model output distribution, compensating for approximations introduced by contextual sparsity. (3) Active weight DRAM-flash swapping pipeline. It orchestrates the DRAM space allocation among the hot weight cache, preloaded active weights, and computation-involved weights based on available memory. Results show ActiveFlow achieves the performance-cost Pareto frontier compared to existing efficiency optimization methods. |
| title | Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flash |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2504.08378 |