MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909232116269056 |
|---|---|
| author | Liu, Zechun Zhao, Changsheng Iandola, Forrest Lai, Chen Tian, Yuandong Fedorov, Igor Xiong, Yunyang Chang, Ernie Shi, Yangyang Krishnamoorthi, Raghuraman Lai, Liangzhen Chandra, Vikas |
| author_facet | Liu, Zechun Zhao, Changsheng Iandola, Forrest Lai, Chen Tian, Yuandong Fedorov, Igor Xiong, Yunyang Chang, Ernie Shi, Yangyang Krishnamoorthi, Raghuraman Lai, Liangzhen Chandra, Vikas |
| contents | This paper addresses the growing need for efficient large language models (LLMs) on mobile devices, driven by increasing cloud costs and latency concerns. We focus on designing top-quality LLMs with fewer than a billion parameters, a practical choice for mobile deployment. Contrary to prevailing belief emphasizing the pivotal role of data and parameter quantity in determining model quality, our investigation underscores the significance of model architecture for sub-billion scale LLMs. Leveraging deep and thin architectures, coupled with embedding sharing and grouped-query attention mechanisms, we establish a strong baseline network denoted as MobileLLM, which attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M state-of-the-art models. Additionally, we propose an immediate block-wise weight-sharing approach with no increase in model size and only marginal latency overhead. The resultant models, denoted as MobileLLM-LS, demonstrate a further accuracy enhancement of 0.7%/0.8% than MobileLLM 125M/350M. Moreover, MobileLLM model family shows significant improvements compared to previous sub-billion models on chat benchmarks, and demonstrates close correctness to LLaMA-v2 7B in API calling tasks, highlighting the capability of small models for common on-device use cases. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_14905 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases Liu, Zechun Zhao, Changsheng Iandola, Forrest Lai, Chen Tian, Yuandong Fedorov, Igor Xiong, Yunyang Chang, Ernie Shi, Yangyang Krishnamoorthi, Raghuraman Lai, Liangzhen Chandra, Vikas Machine Learning Artificial Intelligence Computation and Language This paper addresses the growing need for efficient large language models (LLMs) on mobile devices, driven by increasing cloud costs and latency concerns. We focus on designing top-quality LLMs with fewer than a billion parameters, a practical choice for mobile deployment. Contrary to prevailing belief emphasizing the pivotal role of data and parameter quantity in determining model quality, our investigation underscores the significance of model architecture for sub-billion scale LLMs. Leveraging deep and thin architectures, coupled with embedding sharing and grouped-query attention mechanisms, we establish a strong baseline network denoted as MobileLLM, which attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M state-of-the-art models. Additionally, we propose an immediate block-wise weight-sharing approach with no increase in model size and only marginal latency overhead. The resultant models, denoted as MobileLLM-LS, demonstrate a further accuracy enhancement of 0.7%/0.8% than MobileLLM 125M/350M. Moreover, MobileLLM model family shows significant improvements compared to previous sub-billion models on chat benchmarks, and demonstrates close correctness to LLaMA-v2 7B in API calling tasks, highlighting the capability of small models for common on-device use cases. |
| title | MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2402.14905 |