NVCiM-PT: An NVCiM-assisted Prompt Tuning Framework for Edge LLMs
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917835517722624 |
|---|---|
| author | Qin, Ruiyang Ren, Pengyu Yan, Zheyu Liu, Liu Liu, Dancheng Nassereldine, Amir Xiong, Jinjun Ni, Kai Hu, Sharon Shi, Yiyu |
| author_facet | Qin, Ruiyang Ren, Pengyu Yan, Zheyu Liu, Liu Liu, Dancheng Nassereldine, Amir Xiong, Jinjun Ni, Kai Hu, Sharon Shi, Yiyu |
| contents | Large Language Models (LLMs) deployed on edge devices, known as edge LLMs, need to continuously fine-tune their model parameters from user-generated data under limited resource constraints. However, most existing learning methods are not applicable for edge LLMs because of their reliance on high resources and low learning capacity. Prompt tuning (PT) has recently emerged as an effective fine-tuning method for edge LLMs by only modifying a small portion of LLM parameters, but it suffers from user domain shifts, resulting in repetitive training and losing resource efficiency. Conventional techniques to address domain shift issues often involve complex neural networks and sophisticated training, which are incompatible for PT for edge LLMs. Therefore, an open research question is how to address domain shift issues for edge LLMs with limited resources. In this paper, we propose a prompt tuning framework for edge LLMs, exploiting the benefits offered by non-volatile computing-in-memory (NVCiM) architectures. We introduce a novel NVCiM-assisted PT framework, where we narrow down the core operations to matrix-matrix multiplication, which can then be accelerated by performing in-situ computation on NVCiM. To the best of our knowledge, this is the first work employing NVCiM to improve the edge LLM PT performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_08244 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | NVCiM-PT: An NVCiM-assisted Prompt Tuning Framework for Edge LLMs Qin, Ruiyang Ren, Pengyu Yan, Zheyu Liu, Liu Liu, Dancheng Nassereldine, Amir Xiong, Jinjun Ni, Kai Hu, Sharon Shi, Yiyu Machine Learning Emerging Technologies Large Language Models (LLMs) deployed on edge devices, known as edge LLMs, need to continuously fine-tune their model parameters from user-generated data under limited resource constraints. However, most existing learning methods are not applicable for edge LLMs because of their reliance on high resources and low learning capacity. Prompt tuning (PT) has recently emerged as an effective fine-tuning method for edge LLMs by only modifying a small portion of LLM parameters, but it suffers from user domain shifts, resulting in repetitive training and losing resource efficiency. Conventional techniques to address domain shift issues often involve complex neural networks and sophisticated training, which are incompatible for PT for edge LLMs. Therefore, an open research question is how to address domain shift issues for edge LLMs with limited resources. In this paper, we propose a prompt tuning framework for edge LLMs, exploiting the benefits offered by non-volatile computing-in-memory (NVCiM) architectures. We introduce a novel NVCiM-assisted PT framework, where we narrow down the core operations to matrix-matrix multiplication, which can then be accelerated by performing in-situ computation on NVCiM. To the best of our knowledge, this is the first work employing NVCiM to improve the edge LLM PT performance. |
| title | NVCiM-PT: An NVCiM-assisted Prompt Tuning Framework for Edge LLMs |
| topic | Machine Learning Emerging Technologies |
| url | https://arxiv.org/abs/2411.08244 |