Scaling Laws for Fine-Grained Mixture of Experts
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909103364767744 |
|---|---|
| author | Krajewski, Jakub Ludziejewski, Jan Adamczewski, Kamil Pióro, Maciej Krutul, Michał Antoniak, Szymon Ciebiera, Kamil Król, Krystian Odrzygóźdź, Tomasz Sankowski, Piotr Cygan, Marek Jaszczur, Sebastian |
| author_facet | Krajewski, Jakub Ludziejewski, Jan Adamczewski, Kamil Pióro, Maciej Krutul, Michał Antoniak, Szymon Ciebiera, Kamil Król, Krystian Odrzygóźdź, Tomasz Sankowski, Piotr Cygan, Marek Jaszczur, Sebastian |
| contents | Mixture of Experts (MoE) models have emerged as a primary solution for reducing the computational cost of Large Language Models. In this work, we analyze their scaling properties, incorporating an expanded range of variables. Specifically, we introduce a new hyperparameter, granularity, whose adjustment enables precise control over the size of the experts. Building on this, we establish scaling laws for fine-grained MoE, taking into account the number of training tokens, model size, and granularity. Leveraging these laws, we derive the optimal training configuration for a given computational budget. Our findings not only show that MoE models consistently outperform dense Transformers but also highlight that the efficiency gap between dense and MoE models widens as we scale up the model size and training budget. Furthermore, we demonstrate that the common practice of setting the size of experts in MoE to mirror the feed-forward layer is not optimal at almost any computational budget. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_07871 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Scaling Laws for Fine-Grained Mixture of Experts Krajewski, Jakub Ludziejewski, Jan Adamczewski, Kamil Pióro, Maciej Krutul, Michał Antoniak, Szymon Ciebiera, Kamil Król, Krystian Odrzygóźdź, Tomasz Sankowski, Piotr Cygan, Marek Jaszczur, Sebastian Machine Learning Artificial Intelligence Computation and Language Mixture of Experts (MoE) models have emerged as a primary solution for reducing the computational cost of Large Language Models. In this work, we analyze their scaling properties, incorporating an expanded range of variables. Specifically, we introduce a new hyperparameter, granularity, whose adjustment enables precise control over the size of the experts. Building on this, we establish scaling laws for fine-grained MoE, taking into account the number of training tokens, model size, and granularity. Leveraging these laws, we derive the optimal training configuration for a given computational budget. Our findings not only show that MoE models consistently outperform dense Transformers but also highlight that the efficiency gap between dense and MoE models widens as we scale up the model size and training budget. Furthermore, we demonstrate that the common practice of setting the size of experts in MoE to mirror the feed-forward layer is not optimal at almost any computational budget. |
| title | Scaling Laws for Fine-Grained Mixture of Experts |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2402.07871 |