Improving Hyperparameter Optimization with Checkpointed Model Weights
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911934879629312 |
|---|---|
| author | Mehta, Nikhil Lorraine, Jonathan Masson, Steve Arunachalam, Ramanathan Bhat, Zaid Pervaiz Lucas, James Zachariah, Arun George |
| author_facet | Mehta, Nikhil Lorraine, Jonathan Masson, Steve Arunachalam, Ramanathan Bhat, Zaid Pervaiz Lucas, James Zachariah, Arun George |
| contents | When training deep learning models, the performance depends largely on the selected hyperparameters. However, hyperparameter optimization (HPO) is often one of the most expensive parts of model design. Classical HPO methods treat this as a black-box optimization problem. However, gray-box HPO methods, which incorporate more information about the setup, have emerged as a promising direction for more efficient optimization. For example, using intermediate loss evaluations to terminate bad selections. In this work, we propose an HPO method for neural networks using logged checkpoints of the trained weights to guide future hyperparameter selections. Our method, Forecasting Model Search (FMS), embeds weights into a Gaussian process deep kernel surrogate model, using a permutation-invariant graph metanetwork to be data-efficient with the logged network weights. To facilitate reproducibility and further research, we open-source our code at https://github.com/NVlabs/forecasting-model-search. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_18630 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Improving Hyperparameter Optimization with Checkpointed Model Weights Mehta, Nikhil Lorraine, Jonathan Masson, Steve Arunachalam, Ramanathan Bhat, Zaid Pervaiz Lucas, James Zachariah, Arun George Machine Learning Artificial Intelligence 68T05 I.2.6; G.1.6; D.2.8 When training deep learning models, the performance depends largely on the selected hyperparameters. However, hyperparameter optimization (HPO) is often one of the most expensive parts of model design. Classical HPO methods treat this as a black-box optimization problem. However, gray-box HPO methods, which incorporate more information about the setup, have emerged as a promising direction for more efficient optimization. For example, using intermediate loss evaluations to terminate bad selections. In this work, we propose an HPO method for neural networks using logged checkpoints of the trained weights to guide future hyperparameter selections. Our method, Forecasting Model Search (FMS), embeds weights into a Gaussian process deep kernel surrogate model, using a permutation-invariant graph metanetwork to be data-efficient with the logged network weights. To facilitate reproducibility and further research, we open-source our code at https://github.com/NVlabs/forecasting-model-search. |
| title | Improving Hyperparameter Optimization with Checkpointed Model Weights |
| topic | Machine Learning Artificial Intelligence 68T05 I.2.6; G.1.6; D.2.8 |
| url | https://arxiv.org/abs/2406.18630 |