OLMoE: Open Mixture-of-Experts Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915177952182272 |
|---|---|
| author | Muennighoff, Niklas Soldaini, Luca Groeneveld, Dirk Lo, Kyle Morrison, Jacob Min, Sewon Shi, Weijia Walsh, Pete Tafjord, Oyvind Lambert, Nathan Gu, Yuling Arora, Shane Bhagia, Akshita Schwenk, Dustin Wadden, David Wettig, Alexander Hui, Binyuan Dettmers, Tim Kiela, Douwe Farhadi, Ali Smith, Noah A. Koh, Pang Wei Singh, Amanpreet Hajishirzi, Hannaneh |
| author_facet | Muennighoff, Niklas Soldaini, Luca Groeneveld, Dirk Lo, Kyle Morrison, Jacob Min, Sewon Shi, Weijia Walsh, Pete Tafjord, Oyvind Lambert, Nathan Gu, Yuling Arora, Shane Bhagia, Akshita Schwenk, Dustin Wadden, David Wettig, Alexander Hui, Binyuan Dettmers, Tim Kiela, Douwe Farhadi, Ali Smith, Noah A. Koh, Pang Wei Singh, Amanpreet Hajishirzi, Hannaneh |
| contents | We introduce OLMoE, a fully open, state-of-the-art language model leveraging sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but uses only 1B per input token. We pretrain it on 5 trillion tokens and further adapt it to create OLMoE-1B-7B-Instruct. Our models outperform all available models with similar active parameters, even surpassing larger ones like Llama2-13B-Chat and DeepSeekMoE-16B. We present various experiments on MoE training, analyze routing in our model showing high specialization, and open-source all aspects of our work: model weights, training data, code, and logs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_02060 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | OLMoE: Open Mixture-of-Experts Language Models Muennighoff, Niklas Soldaini, Luca Groeneveld, Dirk Lo, Kyle Morrison, Jacob Min, Sewon Shi, Weijia Walsh, Pete Tafjord, Oyvind Lambert, Nathan Gu, Yuling Arora, Shane Bhagia, Akshita Schwenk, Dustin Wadden, David Wettig, Alexander Hui, Binyuan Dettmers, Tim Kiela, Douwe Farhadi, Ali Smith, Noah A. Koh, Pang Wei Singh, Amanpreet Hajishirzi, Hannaneh Computation and Language Artificial Intelligence Machine Learning We introduce OLMoE, a fully open, state-of-the-art language model leveraging sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but uses only 1B per input token. We pretrain it on 5 trillion tokens and further adapt it to create OLMoE-1B-7B-Instruct. Our models outperform all available models with similar active parameters, even surpassing larger ones like Llama2-13B-Chat and DeepSeekMoE-16B. We present various experiments on MoE training, analyze routing in our model showing high specialization, and open-source all aspects of our work: model weights, training data, code, and logs. |
| title | OLMoE: Open Mixture-of-Experts Language Models |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2409.02060 |