Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.10056 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909651651526656 |
|---|---|
| author | Liu, Qi Ma, Wanjing |
| author_facet | Liu, Qi Ma, Wanjing |
| contents | In this paper, we identify and analyze a recurring training loss pattern, which we term the \textit{Epochal Sawtooth Phenomenon (ESP)}, commonly observed during training with adaptive gradient-based optimizers, particularly Adam optimizer. This pattern is characterized by a sharp drop in loss at the beginning of each epoch, followed by a gradual increase, resulting in a sawtooth-shaped loss curve. Through empirical observations, we demonstrate that while this effect is most pronounced with Adam, it persists, although less severely, with other optimizers such as RMSProp. We empirically analyze the mechanisms underlying ESP, focusing on key factors such as Adam's $β$ parameters, batch size, data shuffling, and sample replacement. Our analysis shows that ESP arises from adaptive learning rate adjustments controlled by the second moment estimate. Additionally, we identify the ``immediate re-exposure to samples'' effect during data shuffling, which causes the model to learn or memorize more at the beginning of each epoch. We also find that smaller values of $β_2$ exacerbate ESP but can act as a form of regularization. While ESP is not necessarily indicative of overfitting, higher model capacity can amplify the phenomenon. To further support our analysis, we replicate ESP through a high-dimensional quadratic minimization task. We demonstrate that ESP can emerge even in simple optimization scenarios, reinforcing the generality of this pattern. The code for reproducing our experiments is available at https://github.com/qiliuchn/training-loss-pattern. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_10056 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Epochal Sawtooth Phenomenon: Unveiling Training Loss Oscillations in Adam and Other Optimizers Liu, Qi Ma, Wanjing Machine Learning Artificial Intelligence In this paper, we identify and analyze a recurring training loss pattern, which we term the \textit{Epochal Sawtooth Phenomenon (ESP)}, commonly observed during training with adaptive gradient-based optimizers, particularly Adam optimizer. This pattern is characterized by a sharp drop in loss at the beginning of each epoch, followed by a gradual increase, resulting in a sawtooth-shaped loss curve. Through empirical observations, we demonstrate that while this effect is most pronounced with Adam, it persists, although less severely, with other optimizers such as RMSProp. We empirically analyze the mechanisms underlying ESP, focusing on key factors such as Adam's $β$ parameters, batch size, data shuffling, and sample replacement. Our analysis shows that ESP arises from adaptive learning rate adjustments controlled by the second moment estimate. Additionally, we identify the ``immediate re-exposure to samples'' effect during data shuffling, which causes the model to learn or memorize more at the beginning of each epoch. We also find that smaller values of $β_2$ exacerbate ESP but can act as a form of regularization. While ESP is not necessarily indicative of overfitting, higher model capacity can amplify the phenomenon. To further support our analysis, we replicate ESP through a high-dimensional quadratic minimization task. We demonstrate that ESP can emerge even in simple optimization scenarios, reinforcing the generality of this pattern. The code for reproducing our experiments is available at https://github.com/qiliuchn/training-loss-pattern. |
| title | The Epochal Sawtooth Phenomenon: Unveiling Training Loss Oscillations in Adam and Other Optimizers |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2410.10056 |