ANGOFA: Leveraging OFA Embedding Initialization and Synthetic Data for Angolan Language Model
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915984734945280 |
|---|---|
| author | Quinjica, Osvaldo Luamba Adelani, David Ifeoluwa |
| author_facet | Quinjica, Osvaldo Luamba Adelani, David Ifeoluwa |
| contents | In recent years, the development of pre-trained language models (PLMs) has gained momentum, showcasing their capacity to transcend linguistic barriers and facilitate knowledge transfer across diverse languages. However, this progress has predominantly bypassed the inclusion of very-low resource languages, creating a notable void in the multilingual landscape. This paper addresses this gap by introducing four tailored PLMs specifically finetuned for Angolan languages, employing a Multilingual Adaptive Fine-tuning (MAFT) approach. In this paper, we survey the role of informed embedding initialization and synthetic data in enhancing the performance of MAFT models in downstream tasks. We improve baseline over SOTA AfroXLMR-base (developed through MAFT) and OFA (an effective embedding initialization) by 12.3 and 3.8 points respectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_02534 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ANGOFA: Leveraging OFA Embedding Initialization and Synthetic Data for Angolan Language Model Quinjica, Osvaldo Luamba Adelani, David Ifeoluwa Computation and Language Artificial Intelligence In recent years, the development of pre-trained language models (PLMs) has gained momentum, showcasing their capacity to transcend linguistic barriers and facilitate knowledge transfer across diverse languages. However, this progress has predominantly bypassed the inclusion of very-low resource languages, creating a notable void in the multilingual landscape. This paper addresses this gap by introducing four tailored PLMs specifically finetuned for Angolan languages, employing a Multilingual Adaptive Fine-tuning (MAFT) approach. In this paper, we survey the role of informed embedding initialization and synthetic data in enhancing the performance of MAFT models in downstream tasks. We improve baseline over SOTA AfroXLMR-base (developed through MAFT) and OFA (an effective embedding initialization) by 12.3 and 3.8 points respectively. |
| title | ANGOFA: Leveraging OFA Embedding Initialization and Synthetic Data for Angolan Language Model |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2404.02534 |