MobileAIBench: Benchmarking LLMs and LMMs for On-Device Use Cases
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914834943049728 |
|---|---|
| author | Murthy, Rithesh Yang, Liangwei Tan, Juntao Awalgaonkar, Tulika Manoj Zhou, Yilun Heinecke, Shelby Desai, Sachin Wu, Jason Xu, Ran Tan, Sarah Zhang, Jianguo Liu, Zhiwei Kokane, Shirley Liu, Zuxin Zhu, Ming Wang, Huan Xiong, Caiming Savarese, Silvio |
| author_facet | Murthy, Rithesh Yang, Liangwei Tan, Juntao Awalgaonkar, Tulika Manoj Zhou, Yilun Heinecke, Shelby Desai, Sachin Wu, Jason Xu, Ran Tan, Sarah Zhang, Jianguo Liu, Zhiwei Kokane, Shirley Liu, Zuxin Zhu, Ming Wang, Huan Xiong, Caiming Savarese, Silvio |
| contents | The deployment of Large Language Models (LLMs) and Large Multimodal Models (LMMs) on mobile devices has gained significant attention due to the benefits of enhanced privacy, stability, and personalization. However, the hardware constraints of mobile devices necessitate the use of models with fewer parameters and model compression techniques like quantization. Currently, there is limited understanding of quantization's impact on various task performances, including LLM tasks, LMM tasks, and, critically, trust and safety. There is a lack of adequate tools for systematically testing these models on mobile devices. To address these gaps, we introduce MobileAIBench, a comprehensive benchmarking framework for evaluating mobile-optimized LLMs and LMMs. MobileAIBench assesses models across different sizes, quantization levels, and tasks, measuring latency and resource consumption on real devices. Our two-part open-source framework includes a library for running evaluations on desktops and an iOS app for on-device latency and hardware utilization measurements. Our thorough analysis aims to accelerate mobile AI research and deployment by providing insights into the performance and feasibility of deploying LLMs and LMMs on mobile platforms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_10290 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MobileAIBench: Benchmarking LLMs and LMMs for On-Device Use Cases Murthy, Rithesh Yang, Liangwei Tan, Juntao Awalgaonkar, Tulika Manoj Zhou, Yilun Heinecke, Shelby Desai, Sachin Wu, Jason Xu, Ran Tan, Sarah Zhang, Jianguo Liu, Zhiwei Kokane, Shirley Liu, Zuxin Zhu, Ming Wang, Huan Xiong, Caiming Savarese, Silvio Computation and Language Artificial Intelligence Machine Learning The deployment of Large Language Models (LLMs) and Large Multimodal Models (LMMs) on mobile devices has gained significant attention due to the benefits of enhanced privacy, stability, and personalization. However, the hardware constraints of mobile devices necessitate the use of models with fewer parameters and model compression techniques like quantization. Currently, there is limited understanding of quantization's impact on various task performances, including LLM tasks, LMM tasks, and, critically, trust and safety. There is a lack of adequate tools for systematically testing these models on mobile devices. To address these gaps, we introduce MobileAIBench, a comprehensive benchmarking framework for evaluating mobile-optimized LLMs and LMMs. MobileAIBench assesses models across different sizes, quantization levels, and tasks, measuring latency and resource consumption on real devices. Our two-part open-source framework includes a library for running evaluations on desktops and an iOS app for on-device latency and hardware utilization measurements. Our thorough analysis aims to accelerate mobile AI research and deployment by providing insights into the performance and feasibility of deploying LLMs and LMMs on mobile platforms. |
| title | MobileAIBench: Benchmarking LLMs and LMMs for On-Device Use Cases |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2406.10290 |