Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2307.09638 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911921219829760 |
|---|---|
| author | Malviya, Pranshu Mordido, Gonçalo Baratin, Aristide Harikandeh, Reza Babanezhad Huang, Jerry Lacoste-Julien, Simon Pascanu, Razvan Chandar, Sarath |
| author_facet | Malviya, Pranshu Mordido, Gonçalo Baratin, Aristide Harikandeh, Reza Babanezhad Huang, Jerry Lacoste-Julien, Simon Pascanu, Razvan Chandar, Sarath |
| contents | Adaptive gradient-based optimizers, notably Adam, have left their mark in training large-scale deep learning models, offering fast convergence and robustness to hyperparameter settings. However, they often struggle with generalization, attributed to their tendency to converge to sharp minima in the loss landscape. To address this, we propose a new memory-augmented version of Adam that encourages exploration towards flatter minima by incorporating a buffer of critical momentum terms during training. This buffer prompts the optimizer to overshoot beyond narrow minima, promoting exploration. Through comprehensive analysis in simple settings, we illustrate the efficacy of our approach in increasing exploration and bias towards flatter minima. We empirically demonstrate that it can improve model performance for image classification on ImageNet and CIFAR10/100, language modelling on Penn Treebank, and online learning tasks on TinyImageNet and 5-dataset. Our code is available at \url{https://github.com/chandar-lab/CMOptimizer}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2307_09638 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Promoting Exploration in Memory-Augmented Adam using Critical Momenta Malviya, Pranshu Mordido, Gonçalo Baratin, Aristide Harikandeh, Reza Babanezhad Huang, Jerry Lacoste-Julien, Simon Pascanu, Razvan Chandar, Sarath Machine Learning Artificial Intelligence Adaptive gradient-based optimizers, notably Adam, have left their mark in training large-scale deep learning models, offering fast convergence and robustness to hyperparameter settings. However, they often struggle with generalization, attributed to their tendency to converge to sharp minima in the loss landscape. To address this, we propose a new memory-augmented version of Adam that encourages exploration towards flatter minima by incorporating a buffer of critical momentum terms during training. This buffer prompts the optimizer to overshoot beyond narrow minima, promoting exploration. Through comprehensive analysis in simple settings, we illustrate the efficacy of our approach in increasing exploration and bias towards flatter minima. We empirically demonstrate that it can improve model performance for image classification on ImageNet and CIFAR10/100, language modelling on Penn Treebank, and online learning tasks on TinyImageNet and 5-dataset. Our code is available at \url{https://github.com/chandar-lab/CMOptimizer}. |
| title | Promoting Exploration in Memory-Augmented Adam using Critical Momenta |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2307.09638 |