Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909501619175424 |
|---|---|
| author | Ludziejewski, Jan Pióro, Maciej Krajewski, Jakub Stefaniak, Maciej Krutul, Michał Małaśnicki, Jan Cygan, Marek Sankowski, Piotr Adamczewski, Kamil Miłoś, Piotr Jaszczur, Sebastian |
| author_facet | Ludziejewski, Jan Pióro, Maciej Krajewski, Jakub Stefaniak, Maciej Krutul, Michał Małaśnicki, Jan Cygan, Marek Sankowski, Piotr Adamczewski, Kamil Miłoś, Piotr Jaszczur, Sebastian |
| contents | Mixture of Experts (MoE) architectures have significantly increased computational efficiency in both research and real-world applications of large-scale machine learning models. However, their scalability and efficiency under memory constraints remain relatively underexplored. In this work, we present joint scaling laws for dense and MoE models, incorporating key factors such as the number of active parameters, dataset size, and the number of experts. Our findings provide a principled framework for selecting the optimal MoE configuration under fixed memory and compute budgets. Surprisingly, we show that MoE models can be more memory-efficient than dense models, contradicting conventional wisdom. To derive and validate the theoretical predictions of our scaling laws, we conduct over 280 experiments with up to 2.7B active parameters and up to 5B total parameters. These results offer actionable insights for designing and deploying MoE models in practical large-scale training scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_05172 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient Ludziejewski, Jan Pióro, Maciej Krajewski, Jakub Stefaniak, Maciej Krutul, Michał Małaśnicki, Jan Cygan, Marek Sankowski, Piotr Adamczewski, Kamil Miłoś, Piotr Jaszczur, Sebastian Machine Learning Artificial Intelligence Computation and Language Mixture of Experts (MoE) architectures have significantly increased computational efficiency in both research and real-world applications of large-scale machine learning models. However, their scalability and efficiency under memory constraints remain relatively underexplored. In this work, we present joint scaling laws for dense and MoE models, incorporating key factors such as the number of active parameters, dataset size, and the number of experts. Our findings provide a principled framework for selecting the optimal MoE configuration under fixed memory and compute budgets. Surprisingly, we show that MoE models can be more memory-efficient than dense models, contradicting conventional wisdom. To derive and validate the theoretical predictions of our scaling laws, we conduct over 280 experiments with up to 2.7B active parameters and up to 5B total parameters. These results offer actionable insights for designing and deploying MoE models in practical large-scale training scenarios. |
| title | Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2502.05172 |