MaCP: Minimal yet Mighty Adaptation via Hierarchical Cosine Projection
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913981426302976 |
|---|---|
| author | Shen, Yixian Bi, Qi Huang, Jia-Hong Zhu, Hongyi Pimentel, Andy D. Pathania, Anuj |
| author_facet | Shen, Yixian Bi, Qi Huang, Jia-Hong Zhu, Hongyi Pimentel, Andy D. Pathania, Anuj |
| contents | We present a new adaptation method MaCP, Minimal yet Mighty adaptive Cosine Projection, that achieves exceptional performance while requiring minimal parameters and memory for fine-tuning large foundation models. Its general idea is to exploit the superior energy compaction and decorrelation properties of cosine projection to improve both model efficiency and accuracy. Specifically, it projects the weight change from the low-rank adaptation into the discrete cosine space. Then, the weight change is partitioned over different levels of the discrete cosine spectrum, and each partition's most critical frequency components are selected. Extensive experiments demonstrate the effectiveness of MaCP across a wide range of single-modality tasks, including natural language understanding, natural language generation, text summarization, as well as multi-modality tasks such as image classification and video understanding. MaCP consistently delivers superior accuracy, significantly reduced computational complexity, and lower memory requirements compared to existing alternatives. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_09103 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MaCP: Minimal yet Mighty Adaptation via Hierarchical Cosine Projection Shen, Yixian Bi, Qi Huang, Jia-Hong Zhu, Hongyi Pimentel, Andy D. Pathania, Anuj Machine Learning Artificial Intelligence We present a new adaptation method MaCP, Minimal yet Mighty adaptive Cosine Projection, that achieves exceptional performance while requiring minimal parameters and memory for fine-tuning large foundation models. Its general idea is to exploit the superior energy compaction and decorrelation properties of cosine projection to improve both model efficiency and accuracy. Specifically, it projects the weight change from the low-rank adaptation into the discrete cosine space. Then, the weight change is partitioned over different levels of the discrete cosine spectrum, and each partition's most critical frequency components are selected. Extensive experiments demonstrate the effectiveness of MaCP across a wide range of single-modality tasks, including natural language understanding, natural language generation, text summarization, as well as multi-modality tasks such as image classification and video understanding. MaCP consistently delivers superior accuracy, significantly reduced computational complexity, and lower memory requirements compared to existing alternatives. |
| title | MaCP: Minimal yet Mighty Adaptation via Hierarchical Cosine Projection |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2410.09103 |