NusaBERT: Teaching IndoBERT to be Multilingual and Multicultural
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909126927319040 |
|---|---|
| author | Wongso, Wilson Setiawan, David Samuel Limcorn, Steven Joyoadikusumo, Ananto |
| author_facet | Wongso, Wilson Setiawan, David Samuel Limcorn, Steven Joyoadikusumo, Ananto |
| contents | Indonesia's linguistic landscape is remarkably diverse, encompassing over 700 languages and dialects, making it one of the world's most linguistically rich nations. This diversity, coupled with the widespread practice of code-switching and the presence of low-resource regional languages, presents unique challenges for modern pre-trained language models. In response to these challenges, we developed NusaBERT, building upon IndoBERT by incorporating vocabulary expansion and leveraging a diverse multilingual corpus that includes regional languages and dialects. Through rigorous evaluation across a range of benchmarks, NusaBERT demonstrates state-of-the-art performance in tasks involving multiple languages of Indonesia, paving the way for future natural language understanding research for under-represented languages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_01817 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | NusaBERT: Teaching IndoBERT to be Multilingual and Multicultural Wongso, Wilson Setiawan, David Samuel Limcorn, Steven Joyoadikusumo, Ananto Computation and Language Indonesia's linguistic landscape is remarkably diverse, encompassing over 700 languages and dialects, making it one of the world's most linguistically rich nations. This diversity, coupled with the widespread practice of code-switching and the presence of low-resource regional languages, presents unique challenges for modern pre-trained language models. In response to these challenges, we developed NusaBERT, building upon IndoBERT by incorporating vocabulary expansion and leveraging a diverse multilingual corpus that includes regional languages and dialects. Through rigorous evaluation across a range of benchmarks, NusaBERT demonstrates state-of-the-art performance in tasks involving multiple languages of Indonesia, paving the way for future natural language understanding research for under-represented languages. |
| title | NusaBERT: Teaching IndoBERT to be Multilingual and Multicultural |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2403.01817 |