RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917702478594048 |
|---|---|
| author | Husain, Jaavid Aktar Dabre, Raj Kumar, Aswanth Gala, Jay Jayakumar, Thanmay Puduppully, Ratish Kunchukuttan, Anoop |
| author_facet | Husain, Jaavid Aktar Dabre, Raj Kumar, Aswanth Gala, Jay Jayakumar, Thanmay Puduppully, Ratish Kunchukuttan, Anoop |
| contents | This study addresses the challenge of extending Large Language Models (LLMs) to non-English languages that use non-Roman scripts. We propose an approach that utilizes the romanized form of text as an interface for LLMs, hypothesizing that its frequent informal use and shared tokens with English enhance cross-lingual alignment. Our approach involves the continual pretraining of an English LLM like Llama 2 on romanized text of non-English, non-Roman script languages, followed by instruction tuning on romanized data. The results indicate that romanized text not only reduces token fertility by 2x-4x but also matches or outperforms native script representation across various NLU, NLG, and MT tasks. Moreover, the embeddings computed on romanized text exhibit closer alignment with their English translations than those from the native script. Our approach presents a promising direction for leveraging the power of English LLMs in languages traditionally underrepresented in NLP. Our code is available on https://github.com/AI4Bharat/romansetu. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_14280 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanization Husain, Jaavid Aktar Dabre, Raj Kumar, Aswanth Gala, Jay Jayakumar, Thanmay Puduppully, Ratish Kunchukuttan, Anoop Computation and Language Artificial Intelligence This study addresses the challenge of extending Large Language Models (LLMs) to non-English languages that use non-Roman scripts. We propose an approach that utilizes the romanized form of text as an interface for LLMs, hypothesizing that its frequent informal use and shared tokens with English enhance cross-lingual alignment. Our approach involves the continual pretraining of an English LLM like Llama 2 on romanized text of non-English, non-Roman script languages, followed by instruction tuning on romanized data. The results indicate that romanized text not only reduces token fertility by 2x-4x but also matches or outperforms native script representation across various NLU, NLG, and MT tasks. Moreover, the embeddings computed on romanized text exhibit closer alignment with their English translations than those from the native script. Our approach presents a promising direction for leveraging the power of English LLMs in languages traditionally underrepresented in NLP. Our code is available on https://github.com/AI4Bharat/romansetu. |
| title | RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanization |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2401.14280 |