Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915037623353344 |
|---|---|
| author | Khade, Omkar Jagdale, Shruti Phaltankar, Abhishek Takalikar, Gauri Joshi, Raviraj |
| author_facet | Khade, Omkar Jagdale, Shruti Phaltankar, Abhishek Takalikar, Gauri Joshi, Raviraj |
| contents | Large Language Models (LLMs) have demonstrated remarkable multilingual capabilities, yet challenges persist in adapting these models for low-resource languages. In this study, we investigate the effects of Low-Rank Adaptation (LoRA) Parameter-Efficient Fine-Tuning (PEFT) on multilingual Gemma models for Marathi, a language with limited resources. Using a translated Alpaca dataset with 52,000 instruction-response pairs, our findings reveal that while evaluation metrics often show a performance decline post-fine-tuning, manual assessments frequently suggest that the fine-tuned models outperform their original counterparts. The observations indicate improvements in target language generation capabilities but a reduction in reasoning abilities following language adaptation. These results underscore the need for improved evaluation methodologies and the creation of high-quality native datasets to accurately assess language-specific model performance in low-resource settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_18571 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning Khade, Omkar Jagdale, Shruti Phaltankar, Abhishek Takalikar, Gauri Joshi, Raviraj Computation and Language Machine Learning Large Language Models (LLMs) have demonstrated remarkable multilingual capabilities, yet challenges persist in adapting these models for low-resource languages. In this study, we investigate the effects of Low-Rank Adaptation (LoRA) Parameter-Efficient Fine-Tuning (PEFT) on multilingual Gemma models for Marathi, a language with limited resources. Using a translated Alpaca dataset with 52,000 instruction-response pairs, our findings reveal that while evaluation metrics often show a performance decline post-fine-tuning, manual assessments frequently suggest that the fine-tuned models outperform their original counterparts. The observations indicate improvements in target language generation capabilities but a reduction in reasoning abilities following language adaptation. These results underscore the need for improved evaluation methodologies and the creation of high-quality native datasets to accurately assess language-specific model performance in low-resource settings. |
| title | Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2411.18571 |