SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912066957213696 |
|---|---|
| author | Chekalina, Viktoriia Rudenko, Anna Mezentsev, Gleb Mikhalev, Alexander Panchenko, Alexander Oseledets, Ivan |
| author_facet | Chekalina, Viktoriia Rudenko, Anna Mezentsev, Gleb Mikhalev, Alexander Panchenko, Alexander Oseledets, Ivan |
| contents | The performance of Transformer models has been enhanced by increasing the number of parameters and the length of the processed text. Consequently, fine-tuning the entire model becomes a memory-intensive process. High-performance methods for parameter-efficient fine-tuning (PEFT) typically work with Attention blocks and often overlook MLP blocks, which contain about half of the model parameters. We propose a new selective PEFT method, namely SparseGrad, that performs well on MLP blocks. We transfer layer gradients to a space where only about 1\% of the layer's elements remain significant. By converting gradients into a sparse structure, we reduce the number of updated parameters. We apply SparseGrad to fine-tune BERT and RoBERTa for the NLU task and LLaMa-2 for the Question-Answering task. In these experiments, with identical memory requirements, our method outperforms LoRA and MeProp, robust popular state-of-the-art PEFT approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_07383 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers Chekalina, Viktoriia Rudenko, Anna Mezentsev, Gleb Mikhalev, Alexander Panchenko, Alexander Oseledets, Ivan Computation and Language Artificial Intelligence The performance of Transformer models has been enhanced by increasing the number of parameters and the length of the processed text. Consequently, fine-tuning the entire model becomes a memory-intensive process. High-performance methods for parameter-efficient fine-tuning (PEFT) typically work with Attention blocks and often overlook MLP blocks, which contain about half of the model parameters. We propose a new selective PEFT method, namely SparseGrad, that performs well on MLP blocks. We transfer layer gradients to a space where only about 1\% of the layer's elements remain significant. By converting gradients into a sparse structure, we reduce the number of updated parameters. We apply SparseGrad to fine-tune BERT and RoBERTa for the NLU task and LLaMa-2 for the Question-Answering task. In these experiments, with identical memory requirements, our method outperforms LoRA and MeProp, robust popular state-of-the-art PEFT approaches. |
| title | SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2410.07383 |