$μ$LO: Compute-Efficient Meta-Generalization of Learned Optimizers
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912973101989888 |
|---|---|
| author | Thérien, Benjamin Joseph, Charles-Étienne Knyazev, Boris Oyallon, Edouard Rish, Irina Belilovsky, Eugene |
| author_facet | Thérien, Benjamin Joseph, Charles-Étienne Knyazev, Boris Oyallon, Edouard Rish, Irina Belilovsky, Eugene |
| contents | Learned optimizers (LOs) have the potential to significantly reduce the wall-clock training time of neural networks. However, they can struggle to optimize unseen tasks (meta-generalize), especially when training networks wider than those seen during meta-training. To address this, we derive the Maximal Update Parametrization ($μ$P) for two state-of-the-art learned optimizer architectures and propose a simple meta-training recipe for $μ$-parameterized LOs ($μ$LOs). Our empirical evaluation demonstrates that LOs meta-trained with our recipe substantially improve meta-generalization to wider unseen tasks when compared to LOs trained under standard parametrization (SP) using the same compute budget. We also empirically observe that $μ$LOs exhibit unexpectedly improved meta-generalization to deeper networks ($5\times$ meta-training) and surprising generalization to much longer training horizons ($25\times$ meta-training) when compared to SP LOs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_00153 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | $μ$LO: Compute-Efficient Meta-Generalization of Learned Optimizers Thérien, Benjamin Joseph, Charles-Étienne Knyazev, Boris Oyallon, Edouard Rish, Irina Belilovsky, Eugene Machine Learning Learned optimizers (LOs) have the potential to significantly reduce the wall-clock training time of neural networks. However, they can struggle to optimize unseen tasks (meta-generalize), especially when training networks wider than those seen during meta-training. To address this, we derive the Maximal Update Parametrization ($μ$P) for two state-of-the-art learned optimizer architectures and propose a simple meta-training recipe for $μ$-parameterized LOs ($μ$LOs). Our empirical evaluation demonstrates that LOs meta-trained with our recipe substantially improve meta-generalization to wider unseen tasks when compared to LOs trained under standard parametrization (SP) using the same compute budget. We also empirically observe that $μ$LOs exhibit unexpectedly improved meta-generalization to deeper networks ($5\times$ meta-training) and surprising generalization to much longer training horizons ($25\times$ meta-training) when compared to SP LOs. |
| title | $μ$LO: Compute-Efficient Meta-Generalization of Learned Optimizers |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2406.00153 |