Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912057086967808 |
|---|---|
| author | Iyer, Vivek Malik, Bhavitvya Stepachev, Pavel Chen, Pinzhen Haddow, Barry Birch, Alexandra |
| author_facet | Iyer, Vivek Malik, Bhavitvya Stepachev, Pavel Chen, Pinzhen Haddow, Barry Birch, Alexandra |
| contents | Despite the recent popularity of Large Language Models (LLMs) in Machine Translation (MT), their performance in low-resource languages (LRLs) still lags significantly behind Neural Machine Translation (NMT) models. In this work, we explore what it would take to adapt LLMs for the low-resource setting. Particularly, we re-examine the role of two factors: a) the importance and application of parallel data, and b) diversity in Supervised Fine-Tuning (SFT). Recently, parallel data has seen reduced use in adapting LLMs for MT, while data diversity has been embraced to promote transfer across languages and tasks. However, for low-resource LLM-MT, we show that the opposite is true for both considerations: a) parallel data is critical during both pre-training and SFT; b) diversity tends to cause interference instead of transfer. Our experiments with three LLMs across two low-resourced language groups -- Indigenous American and North-East Indian -- reveal consistent trends, underscoring the generalizability of our findings. We believe these insights will be valuable for scaling to massively multilingual LLM-MT models that can effectively serve LRLs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_12780 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translation Iyer, Vivek Malik, Bhavitvya Stepachev, Pavel Chen, Pinzhen Haddow, Barry Birch, Alexandra Computation and Language Despite the recent popularity of Large Language Models (LLMs) in Machine Translation (MT), their performance in low-resource languages (LRLs) still lags significantly behind Neural Machine Translation (NMT) models. In this work, we explore what it would take to adapt LLMs for the low-resource setting. Particularly, we re-examine the role of two factors: a) the importance and application of parallel data, and b) diversity in Supervised Fine-Tuning (SFT). Recently, parallel data has seen reduced use in adapting LLMs for MT, while data diversity has been embraced to promote transfer across languages and tasks. However, for low-resource LLM-MT, we show that the opposite is true for both considerations: a) parallel data is critical during both pre-training and SFT; b) diversity tends to cause interference instead of transfer. Our experiments with three LLMs across two low-resourced language groups -- Indigenous American and North-East Indian -- reveal consistent trends, underscoring the generalizability of our findings. We believe these insights will be valuable for scaling to massively multilingual LLM-MT models that can effectively serve LRLs. |
| title | Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2408.12780 |