$μ$-Parametrization for Mixture of Experts
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918157384417280 |
|---|---|
| author | Małaśnicki, Jan Ciebiera, Kamil Boruń, Mateusz Pióro, Maciej Ludziejewski, Jan Stefaniak, Maciej Krutul, Michał Jaszczur, Sebastian Cygan, Marek Adamczewski, Kamil Krajewski, Jakub |
| author_facet | Małaśnicki, Jan Ciebiera, Kamil Boruń, Mateusz Pióro, Maciej Ludziejewski, Jan Stefaniak, Maciej Krutul, Michał Jaszczur, Sebastian Cygan, Marek Adamczewski, Kamil Krajewski, Jakub |
| contents | Recent years have seen a growing interest and adoption of LLMs, with Mixture-of-Experts (MoE) emerging as a leading architecture in extremely large models. Currently, the largest open-source models reach over $1$T parameters. At such scales, hyperparameter tuning becomes prohibitively expensive. Precisely for this reason, the $μ$Transfer is becoming a key technique. It allows for seamless transfer of optimal hyperparameters across model scales, resulting in a huge reduction in tuning costs. However, existing work has primarily focused on dense LLMs, leaving MoE architectures unexplored. In this work, we derive a $μ$-Parameterization for MoE, providing theoretical guarantees for feature learning across model widths. Our experiments demonstrate that the optimal learning rate reliably transfers across model sizes, establishing a foundation for efficient hyperparameter tuning in large-scale MoE models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_09752 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | $μ$-Parametrization for Mixture of Experts Małaśnicki, Jan Ciebiera, Kamil Boruń, Mateusz Pióro, Maciej Ludziejewski, Jan Stefaniak, Maciej Krutul, Michał Jaszczur, Sebastian Cygan, Marek Adamczewski, Kamil Krajewski, Jakub Machine Learning Recent years have seen a growing interest and adoption of LLMs, with Mixture-of-Experts (MoE) emerging as a leading architecture in extremely large models. Currently, the largest open-source models reach over $1$T parameters. At such scales, hyperparameter tuning becomes prohibitively expensive. Precisely for this reason, the $μ$Transfer is becoming a key technique. It allows for seamless transfer of optimal hyperparameters across model scales, resulting in a huge reduction in tuning costs. However, existing work has primarily focused on dense LLMs, leaving MoE architectures unexplored. In this work, we derive a $μ$-Parameterization for MoE, providing theoretical guarantees for feature learning across model widths. Our experiments demonstrate that the optimal learning rate reliably transfers across model sizes, establishing a foundation for efficient hyperparameter tuning in large-scale MoE models. |
| title | $μ$-Parametrization for Mixture of Experts |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2508.09752 |