On-Device LLMs for SMEs: Challenges and Opportunities
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910659850010624 |
|---|---|
| author | Yee, Jeremy Stephen Gabriel Ng, Pai Chet Wang, Zhengkui McLoughlin, Ian Ng, Aik Beng See, Simon |
| author_facet | Yee, Jeremy Stephen Gabriel Ng, Pai Chet Wang, Zhengkui McLoughlin, Ian Ng, Aik Beng See, Simon |
| contents | This paper presents a systematic review of the infrastructure requirements for deploying Large Language Models (LLMs) on-device within the context of small and medium-sized enterprises (SMEs), focusing on both hardware and software perspectives. From the hardware viewpoint, we discuss the utilization of processing units like GPUs and TPUs, efficient memory and storage solutions, and strategies for effective deployment, addressing the challenges of limited computational resources typical in SME settings. From the software perspective, we explore framework compatibility, operating system optimization, and the use of specialized libraries tailored for resource-constrained environments. The review is structured to first identify the unique challenges faced by SMEs in deploying LLMs on-device, followed by an exploration of the opportunities that both hardware innovations and software adaptations offer to overcome these obstacles. Such a structured review provides practical insights, contributing significantly to the community by enhancing the technological resilience of SMEs in integrating LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_16070 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On-Device LLMs for SMEs: Challenges and Opportunities Yee, Jeremy Stephen Gabriel Ng, Pai Chet Wang, Zhengkui McLoughlin, Ian Ng, Aik Beng See, Simon Artificial Intelligence Computation and Language 68T07 I.2 This paper presents a systematic review of the infrastructure requirements for deploying Large Language Models (LLMs) on-device within the context of small and medium-sized enterprises (SMEs), focusing on both hardware and software perspectives. From the hardware viewpoint, we discuss the utilization of processing units like GPUs and TPUs, efficient memory and storage solutions, and strategies for effective deployment, addressing the challenges of limited computational resources typical in SME settings. From the software perspective, we explore framework compatibility, operating system optimization, and the use of specialized libraries tailored for resource-constrained environments. The review is structured to first identify the unique challenges faced by SMEs in deploying LLMs on-device, followed by an exploration of the opportunities that both hardware innovations and software adaptations offer to overcome these obstacles. Such a structured review provides practical insights, contributing significantly to the community by enhancing the technological resilience of SMEs in integrating LLMs. |
| title | On-Device LLMs for SMEs: Challenges and Opportunities |
| topic | Artificial Intelligence Computation and Language 68T07 I.2 |
| url | https://arxiv.org/abs/2410.16070 |