LOLA -- An Open-Source Massively Multilingual Large Language Model
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910809558351872 |
|---|---|
| author | Srivastava, Nikit Kuchelev, Denis Ngoli, Tatiana Moteu Shetty, Kshitij Röder, Michael Zahera, Hamada Moussallem, Diego Ngomo, Axel-Cyrille Ngonga |
| author_facet | Srivastava, Nikit Kuchelev, Denis Ngoli, Tatiana Moteu Shetty, Kshitij Röder, Michael Zahera, Hamada Moussallem, Diego Ngomo, Axel-Cyrille Ngonga |
| contents | This paper presents LOLA, a massively multilingual large language model trained on more than 160 languages using a sparse Mixture-of-Experts Transformer architecture. Our architectural and implementation choices address the challenge of harnessing linguistic diversity while maintaining efficiency and avoiding the common pitfalls of multilinguality. Our analysis of the evaluation results shows competitive performance in natural language generation and understanding tasks. Additionally, we demonstrate how the learned expert-routing mechanism exploits implicit phylogenetic linguistic patterns to potentially alleviate the curse of multilinguality. We provide an in-depth look at the training process, an analysis of the datasets, and a balanced exploration of the model's strengths and limitations. As an open-source model, LOLA promotes reproducibility and serves as a robust foundation for future research. Our findings enable the development of compute-efficient multilingual models with strong, scalable performance across languages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_11272 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LOLA -- An Open-Source Massively Multilingual Large Language Model Srivastava, Nikit Kuchelev, Denis Ngoli, Tatiana Moteu Shetty, Kshitij Röder, Michael Zahera, Hamada Moussallem, Diego Ngomo, Axel-Cyrille Ngonga Computation and Language Artificial Intelligence Machine Learning This paper presents LOLA, a massively multilingual large language model trained on more than 160 languages using a sparse Mixture-of-Experts Transformer architecture. Our architectural and implementation choices address the challenge of harnessing linguistic diversity while maintaining efficiency and avoiding the common pitfalls of multilinguality. Our analysis of the evaluation results shows competitive performance in natural language generation and understanding tasks. Additionally, we demonstrate how the learned expert-routing mechanism exploits implicit phylogenetic linguistic patterns to potentially alleviate the curse of multilinguality. We provide an in-depth look at the training process, an analysis of the datasets, and a balanced exploration of the model's strengths and limitations. As an open-source model, LOLA promotes reproducibility and serves as a robust foundation for future research. Our findings enable the development of compute-efficient multilingual models with strong, scalable performance across languages. |
| title | LOLA -- An Open-Source Massively Multilingual Large Language Model |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2409.11272 |