Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910639674359808 |
|---|---|
| author | Blevins, Terra Limisiewicz, Tomasz Gururangan, Suchin Li, Margaret Gonen, Hila Smith, Noah A. Zettlemoyer, Luke |
| author_facet | Blevins, Terra Limisiewicz, Tomasz Gururangan, Suchin Li, Margaret Gonen, Hila Smith, Noah A. Zettlemoyer, Luke |
| contents | Despite their popularity in non-English NLP, multilingual language models often underperform monolingual ones due to inter-language competition for model parameters. We propose Cross-lingual Expert Language Models (X-ELM), which mitigate this competition by independently training language models on subsets of the multilingual corpus. This process specializes X-ELMs to different languages while remaining effective as a multilingual ensemble. Our experiments show that when given the same compute budget, X-ELM outperforms jointly trained multilingual models across all considered languages and that these gains transfer to downstream tasks. X-ELM provides additional benefits over performance improvements: new experts can be iteratively added, adapting X-ELM to new languages without catastrophic forgetting. Furthermore, training is asynchronous, reducing the hardware requirements for multilingual training and democratizing multilingual modeling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_10440 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models Blevins, Terra Limisiewicz, Tomasz Gururangan, Suchin Li, Margaret Gonen, Hila Smith, Noah A. Zettlemoyer, Luke Computation and Language Despite their popularity in non-English NLP, multilingual language models often underperform monolingual ones due to inter-language competition for model parameters. We propose Cross-lingual Expert Language Models (X-ELM), which mitigate this competition by independently training language models on subsets of the multilingual corpus. This process specializes X-ELMs to different languages while remaining effective as a multilingual ensemble. Our experiments show that when given the same compute budget, X-ELM outperforms jointly trained multilingual models across all considered languages and that these gains transfer to downstream tasks. X-ELM provides additional benefits over performance improvements: new experts can be iteratively added, adapting X-ELM to new languages without catastrophic forgetting. Furthermore, training is asynchronous, reducing the hardware requirements for multilingual training and democratizing multilingual modeling. |
| title | Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2401.10440 |