Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.12847 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917807866773504 |
|---|---|
| author | Lin, Yu-Chen Li, Wei-Hua Chen, Jun-Cheng Chen, Chu-Song |
| author_facet | Lin, Yu-Chen Li, Wei-Hua Chen, Jun-Cheng Chen, Chu-Song |
| contents | Prompt Tuning has been a popular Parameter-Efficient Fine-Tuning method attributed to its remarkable performance with few updated parameters on various large-scale pretrained Language Models (PLMs). Traditionally, each prompt has been considered indivisible and updated independently, leading the parameters increase proportionally as prompt length grows. To address this issue, we propose Adaptive Codebook for Composite and Efficient Prompt Tuning (ACCEPT). In our method, we refer to the concept of product quantization (PQ), allowing all soft prompts to share a set of learnable codebook vectors in each subspace, with each prompt differentiated by a set of adaptive weights. We achieve the superior performance on 17 diverse natural language tasks including natural language understanding (NLU) and question answering (QA) tasks by tuning only 0.3% of parameters of the PLMs. Our approach also excels in few-shot and large model settings, highlighting its significant potential. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_12847 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ACCEPT: Adaptive Codebook for Composite and Efficient Prompt Tuning Lin, Yu-Chen Li, Wei-Hua Chen, Jun-Cheng Chen, Chu-Song Computation and Language Artificial Intelligence Prompt Tuning has been a popular Parameter-Efficient Fine-Tuning method attributed to its remarkable performance with few updated parameters on various large-scale pretrained Language Models (PLMs). Traditionally, each prompt has been considered indivisible and updated independently, leading the parameters increase proportionally as prompt length grows. To address this issue, we propose Adaptive Codebook for Composite and Efficient Prompt Tuning (ACCEPT). In our method, we refer to the concept of product quantization (PQ), allowing all soft prompts to share a set of learnable codebook vectors in each subspace, with each prompt differentiated by a set of adaptive weights. We achieve the superior performance on 17 diverse natural language tasks including natural language understanding (NLU) and question answering (QA) tasks by tuning only 0.3% of parameters of the PLMs. Our approach also excels in few-shot and large model settings, highlighting its significant potential. |
| title | ACCEPT: Adaptive Codebook for Composite and Efficient Prompt Tuning |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2410.12847 |