MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale Deployment
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917441818329088 |
|---|---|
| author | Huang, Hanxian Fedorov, Igor Gromov, Andrey Beckerman, Bernard Suda, Naveen Eriksson, David Balandat, Maximilian Conway, Rylan Huber, Patrick Sankar, Chinnadhurai Dalmia, Ayushi Liu, Zechun Wu, Lemeng Elgamal, Tarek Sagar, Adithya Chandra, Vikas Krishnamoorthi, Raghuraman |
| author_facet | Huang, Hanxian Fedorov, Igor Gromov, Andrey Beckerman, Bernard Suda, Naveen Eriksson, David Balandat, Maximilian Conway, Rylan Huber, Patrick Sankar, Chinnadhurai Dalmia, Ayushi Liu, Zechun Wu, Lemeng Elgamal, Tarek Sagar, Adithya Chandra, Vikas Krishnamoorthi, Raghuraman |
| contents | Real-time AI experiences call for on-device large language models (OD-LLMs) optimized for efficient deployment on resource-constrained hardware. The most useful OD-LLMs produce near-real-time responses and exhibit broad hardware compatibility, maximizing user reach. We present a methodology for designing such models using hardware-in-the-loop architecture search under mobile latency constraints. This system is amenable to industry-scale deployment: it generates models deployable without custom kernels and compatible with standard mobile runtimes like Executorch. Our methodology avoids specialized attention mechanisms and instead uses attention skipping for long-context acceleration.
Our approach jointly optimizes model architecture (layers, dimensions) and attention pattern. To efficiently evaluate candidates, we treat each as a pruned version of a pretrained backbone with inherited weights, thereby achieving high accuracy with minimal continued pretraining. We leverage the low cost of latency evaluation in a staged process: learning an accurate latency model first, then searching for the Pareto-frontier across latency and quality.
This yields MobileLLM-Flash, a family of foundation models (350M, 650M, 1.4B) for efficient on-device use with strong capabilities, supporting up to 8k context length. MobileLLM-Flash delivers up to 1.8x and 1.6x faster prefill and decode on mobile CPUs with comparable or superior quality. Our analysis of Pareto-frontier design choices offers actionable principles for OD-LLM design. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_15954 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale Deployment Huang, Hanxian Fedorov, Igor Gromov, Andrey Beckerman, Bernard Suda, Naveen Eriksson, David Balandat, Maximilian Conway, Rylan Huber, Patrick Sankar, Chinnadhurai Dalmia, Ayushi Liu, Zechun Wu, Lemeng Elgamal, Tarek Sagar, Adithya Chandra, Vikas Krishnamoorthi, Raghuraman Machine Learning Artificial Intelligence Real-time AI experiences call for on-device large language models (OD-LLMs) optimized for efficient deployment on resource-constrained hardware. The most useful OD-LLMs produce near-real-time responses and exhibit broad hardware compatibility, maximizing user reach. We present a methodology for designing such models using hardware-in-the-loop architecture search under mobile latency constraints. This system is amenable to industry-scale deployment: it generates models deployable without custom kernels and compatible with standard mobile runtimes like Executorch. Our methodology avoids specialized attention mechanisms and instead uses attention skipping for long-context acceleration. Our approach jointly optimizes model architecture (layers, dimensions) and attention pattern. To efficiently evaluate candidates, we treat each as a pruned version of a pretrained backbone with inherited weights, thereby achieving high accuracy with minimal continued pretraining. We leverage the low cost of latency evaluation in a staged process: learning an accurate latency model first, then searching for the Pareto-frontier across latency and quality. This yields MobileLLM-Flash, a family of foundation models (350M, 650M, 1.4B) for efficient on-device use with strong capabilities, supporting up to 8k context length. MobileLLM-Flash delivers up to 1.8x and 1.6x faster prefill and decode on mobile CPUs with comparable or superior quality. Our analysis of Pareto-frontier design choices offers actionable principles for OD-LLM design. |
| title | MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale Deployment |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2603.15954 |