BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2021
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912858416087040 |
|---|---|
| author | Ben-Zaken, Elad Ravfogel, Shauli Goldberg, Yoav |
| author_facet | Ben-Zaken, Elad Ravfogel, Shauli Goldberg, Yoav |
| contents | We introduce BitFit, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified. We show that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. For larger data, the method is competitive with other sparse fine-tuning methods. Besides their practical utility, these findings are relevant for the question of understanding the commonly-used process of finetuning: they support the hypothesis that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2106_10199 |
| institution | arXiv |
| publishDate | 2021 |
| record_format | arxiv |
| spellingShingle | BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models Ben-Zaken, Elad Ravfogel, Shauli Goldberg, Yoav Machine Learning Computation and Language We introduce BitFit, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified. We show that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. For larger data, the method is competitive with other sparse fine-tuning methods. Besides their practical utility, these findings are relevant for the question of understanding the commonly-used process of finetuning: they support the hypothesis that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge. |
| title | BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2106.10199 |