Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911828622180352 |
|---|---|
| author | Su, Tong Peng, Xin Thillainathan, Sarubi Guzmán, David Ranathunga, Surangika Lee, En-Shiun Annie |
| author_facet | Su, Tong Peng, Xin Thillainathan, Sarubi Guzmán, David Ranathunga, Surangika Lee, En-Shiun Annie |
| contents | Parameter-efficient fine-tuning (PEFT) methods are increasingly vital in adapting large-scale pre-trained language models for diverse tasks, offering a balance between adaptability and computational efficiency. They are important in Low-Resource Language (LRL) Neural Machine Translation (NMT) to enhance translation accuracy with minimal resources. However, their practical effectiveness varies significantly across different languages. We conducted comprehensive empirical experiments with varying LRL domains and sizes to evaluate the performance of 8 PEFT methods with in total of 15 architectures using the SacreBLEU score. We showed that 6 PEFT architectures outperform the baseline for both in-domain and out-domain tests and the Houlsby+Inversion adapter has the best performance overall, proving the effectiveness of PEFT methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_04212 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translation Su, Tong Peng, Xin Thillainathan, Sarubi Guzmán, David Ranathunga, Surangika Lee, En-Shiun Annie Computation and Language Parameter-efficient fine-tuning (PEFT) methods are increasingly vital in adapting large-scale pre-trained language models for diverse tasks, offering a balance between adaptability and computational efficiency. They are important in Low-Resource Language (LRL) Neural Machine Translation (NMT) to enhance translation accuracy with minimal resources. However, their practical effectiveness varies significantly across different languages. We conducted comprehensive empirical experiments with varying LRL domains and sizes to evaluate the performance of 8 PEFT methods with in total of 15 architectures using the SacreBLEU score. We showed that 6 PEFT architectures outperform the baseline for both in-domain and out-domain tests and the Houlsby+Inversion adapter has the best performance overall, proving the effectiveness of PEFT methods. |
| title | Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2404.04212 |