Comparison between parameter-efficient techniques and full fine-tuning: A case study on multilingual news article classification
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910400688160768 |
|---|---|
| author | Razuvayevskaya, Olesya Wu, Ben Leite, Joao A. Heppell, Freddy Srba, Ivan Scarton, Carolina Bontcheva, Kalina Song, Xingyi |
| author_facet | Razuvayevskaya, Olesya Wu, Ben Leite, Joao A. Heppell, Freddy Srba, Ivan Scarton, Carolina Bontcheva, Kalina Song, Xingyi |
| contents | Adapters and Low-Rank Adaptation (LoRA) are parameter-efficient fine-tuning techniques designed to make the training of language models more efficient. Previous results demonstrated that these methods can even improve performance on some classification tasks. This paper complements the existing research by investigating how these techniques influence the classification performance and computation costs compared to full fine-tuning when applied to multilingual text classification tasks (genre, framing, and persuasion techniques detection; with different input lengths, number of predicted classes and classification difficulty), some of which have limited training data. In addition, we conduct in-depth analyses of their efficacy across different training scenarios (training on the original multilingual data; on the translations into English; and on a subset of English-only data) and different languages. Our findings provide valuable insights into the applicability of the parameter-efficient fine-tuning techniques, particularly to complex multilingual and multilabel classification tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2308_07282 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Comparison between parameter-efficient techniques and full fine-tuning: A case study on multilingual news article classification Razuvayevskaya, Olesya Wu, Ben Leite, Joao A. Heppell, Freddy Srba, Ivan Scarton, Carolina Bontcheva, Kalina Song, Xingyi Computation and Language Adapters and Low-Rank Adaptation (LoRA) are parameter-efficient fine-tuning techniques designed to make the training of language models more efficient. Previous results demonstrated that these methods can even improve performance on some classification tasks. This paper complements the existing research by investigating how these techniques influence the classification performance and computation costs compared to full fine-tuning when applied to multilingual text classification tasks (genre, framing, and persuasion techniques detection; with different input lengths, number of predicted classes and classification difficulty), some of which have limited training data. In addition, we conduct in-depth analyses of their efficacy across different training scenarios (training on the original multilingual data; on the translations into English; and on a subset of English-only data) and different languages. Our findings provide valuable insights into the applicability of the parameter-efficient fine-tuning techniques, particularly to complex multilingual and multilabel classification tasks. |
| title | Comparison between parameter-efficient techniques and full fine-tuning: A case study on multilingual news article classification |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2308.07282 |