Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.07494 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910015306072064 |
|---|---|
| author | Wu, Shenxi Zhang, Haosong Ma, Xingjian Bian, Shirui Zhang, Yichi Chen, Xi Lin, Wei |
| author_facet | Wu, Shenxi Zhang, Haosong Ma, Xingjian Bian, Shirui Zhang, Yichi Chen, Xi Lin, Wei |
| contents | Deeper modern architectures are costly to train, making hyperparameter transfer preferable to expensive repeated tuning. Maximal Update Parametrization ($μ$P) helps explain why many hyperparameters transfer across width. Yet depth scaling is less understood for modern architectures, whose computation graphs contain multiple parallel paths and residual aggregation. To unify various non-recurrent multi-path neural networks such as CNNs, ResNets, and Transformers, we introduce a graph-based notion of effective depth. Under stabilizing initializations and a maximal-update criterion, we show that the optimal learning rate decays with effective depth following a universal -3/2 power law. Here, the maximal-update criterion maximizes the typical one-step representation change at initialization without causing instability, and effective depth is the minimal path length from input to output, counting layers and residual additions. Experiments across diverse architectures confirm the predicted slope and enable reliable zero-shot transfer of learning rates across depths and widths, turning depth scaling into a predictable hyperparameter-transfer problem. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_07494 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networks Wu, Shenxi Zhang, Haosong Ma, Xingjian Bian, Shirui Zhang, Yichi Chen, Xi Lin, Wei Machine Learning Deeper modern architectures are costly to train, making hyperparameter transfer preferable to expensive repeated tuning. Maximal Update Parametrization ($μ$P) helps explain why many hyperparameters transfer across width. Yet depth scaling is less understood for modern architectures, whose computation graphs contain multiple parallel paths and residual aggregation. To unify various non-recurrent multi-path neural networks such as CNNs, ResNets, and Transformers, we introduce a graph-based notion of effective depth. Under stabilizing initializations and a maximal-update criterion, we show that the optimal learning rate decays with effective depth following a universal -3/2 power law. Here, the maximal-update criterion maximizes the typical one-step representation change at initialization without causing instability, and effective depth is the minimal path length from input to output, counting layers and residual additions. Experiments across diverse architectures confirm the predicted slope and enable reliable zero-shot transfer of learning rates across depths and widths, turning depth scaling into a predictable hyperparameter-transfer problem. |
| title | Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networks |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2602.07494 |