Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.10443 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913890069118976 |
|---|---|
| author | Wang, Zhaode Yang, Jingbang Qian, Xinyu Xing, Shiwen Jiang, Xiaotang Lv, Chengfei Zhang, Shengyu |
| author_facet | Wang, Zhaode Yang, Jingbang Qian, Xinyu Xing, Shiwen Jiang, Xiaotang Lv, Chengfei Zhang, Shengyu |
| contents | Large language models (LLMs) have demonstrated exceptional performance across a variety of tasks. However, their substantial scale leads to significant computational resource consumption during inference, resulting in high costs. Consequently, edge device inference presents a promising solution. The primary challenges of edge inference include memory usage and inference speed. This paper introduces MNN-LLM, a framework specifically designed to accelerate the deployment of large language models on mobile devices. MNN-LLM addresses the runtime characteristics of LLMs through model quantization and DRAM-Flash hybrid storage, effectively reducing memory usage. It rearranges weights and inputs based on mobile CPU instruction sets and GPU characteristics while employing strategies such as multicore load balancing, mixed-precision floating-point operations, and geometric computations to enhance performance. Notably, MNN-LLM achieves up to a 8.6x speed increase compared to current mainstream LLM-specific frameworks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_10443 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devices Wang, Zhaode Yang, Jingbang Qian, Xinyu Xing, Shiwen Jiang, Xiaotang Lv, Chengfei Zhang, Shengyu Machine Learning Large language models (LLMs) have demonstrated exceptional performance across a variety of tasks. However, their substantial scale leads to significant computational resource consumption during inference, resulting in high costs. Consequently, edge device inference presents a promising solution. The primary challenges of edge inference include memory usage and inference speed. This paper introduces MNN-LLM, a framework specifically designed to accelerate the deployment of large language models on mobile devices. MNN-LLM addresses the runtime characteristics of LLMs through model quantization and DRAM-Flash hybrid storage, effectively reducing memory usage. It rearranges weights and inputs based on mobile CPU instruction sets and GPU characteristics while employing strategies such as multicore load balancing, mixed-precision floating-point operations, and geometric computations to enhance performance. Notably, MNN-LLM achieves up to a 8.6x speed increase compared to current mainstream LLM-specific frameworks. |
| title | MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devices |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2506.10443 |