Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.15290 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910222585430016 |
|---|---|
| author | Chickering, Kyle R. Wang, Huijuan Wu, Mengxi Moreno, Alexander Chen, Muhao Ma, Xuezhe Soboleva, Daria Hestness, Joel Liu, Zhengzhong Xing, Eric |
| author_facet | Chickering, Kyle R. Wang, Huijuan Wu, Mengxi Moreno, Alexander Chen, Muhao Ma, Xuezhe Soboleva, Daria Hestness, Joel Liu, Zhengzhong Xing, Eric |
| contents | Hyperparameter transfer across model architectures dramatically reduces the amount of compute necessary for tuning large language models (LLMs). The maximal update parameterization (μP) ensures transfer through principled mathematical analysis but can be challenging to derive for new model architectures. Building on the spectral feature-learning view of Yang et al. (2023a), we make two advances. First, we promote spectral norm conditions on the weights from a heuristic to the definition of feature learning, and as a consequence arrive at the Complete-P depth and weight-decay scalings without recourse to lazy-learning. Second, we consider a modified spectral norm that preserves the valid scaling law of network weights when weight matrices are not full rank. This enables (to our knowledge, the first) derivation of μP scalings for grouped-query attention (GQA). We demonstrate the efficacy of our theoretical derivations by showing learning rate transfer across the GQA repetition hyperparameter as well as experiments regarding transfer over weight decay. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_15290 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | GQA-μP: The maximal parameterization update for grouped query attention Chickering, Kyle R. Wang, Huijuan Wu, Mengxi Moreno, Alexander Chen, Muhao Ma, Xuezhe Soboleva, Daria Hestness, Joel Liu, Zhengzhong Xing, Eric Machine Learning Artificial Intelligence Hyperparameter transfer across model architectures dramatically reduces the amount of compute necessary for tuning large language models (LLMs). The maximal update parameterization (μP) ensures transfer through principled mathematical analysis but can be challenging to derive for new model architectures. Building on the spectral feature-learning view of Yang et al. (2023a), we make two advances. First, we promote spectral norm conditions on the weights from a heuristic to the definition of feature learning, and as a consequence arrive at the Complete-P depth and weight-decay scalings without recourse to lazy-learning. Second, we consider a modified spectral norm that preserves the valid scaling law of network weights when weight matrices are not full rank. This enables (to our knowledge, the first) derivation of μP scalings for grouped-query attention (GQA). We demonstrate the efficacy of our theoretical derivations by showing learning rate transfer across the GQA repetition hyperparameter as well as experiments regarding transfer over weight decay. |
| title | GQA-μP: The maximal parameterization update for grouped query attention |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2605.15290 |