On the Performance Analysis of Momentum Method: A Frequency Domain Perspective
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916748800819200 |
|---|---|
| author | Li, Xianliang Luo, Jun Zheng, Zhiwei Wang, Hanxiao Luo, Li Wen, Lingkun Wu, Linlong Xu, Sheng |
| author_facet | Li, Xianliang Luo, Jun Zheng, Zhiwei Wang, Hanxiao Luo, Li Wen, Lingkun Wu, Linlong Xu, Sheng |
| contents | Momentum-based optimizers are widely adopted for training neural networks. However, the optimal selection of momentum coefficients remains elusive. This uncertainty impedes a clear understanding of the role of momentum in stochastic gradient methods. In this paper, we present a frequency domain analysis framework that interprets the momentum method as a time-variant filter for gradients, where adjustments to momentum coefficients modify the filter characteristics. Our experiments support this perspective and provide a deeper understanding of the mechanism involved. Moreover, our analysis reveals the following significant findings: high-frequency gradient components are undesired in the late stages of training; preserving the original gradient in the early stages, and gradually amplifying low-frequency gradient components during training both enhance performance. Based on these insights, we propose Frequency Stochastic Gradient Descent with Momentum (FSGDM), a heuristic optimizer that dynamically adjusts the momentum filtering characteristic with an empirically effective dynamic magnitude response. Experimental results demonstrate the superiority of FSGDM over conventional momentum optimizers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_19671 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On the Performance Analysis of Momentum Method: A Frequency Domain Perspective Li, Xianliang Luo, Jun Zheng, Zhiwei Wang, Hanxiao Luo, Li Wen, Lingkun Wu, Linlong Xu, Sheng Machine Learning Momentum-based optimizers are widely adopted for training neural networks. However, the optimal selection of momentum coefficients remains elusive. This uncertainty impedes a clear understanding of the role of momentum in stochastic gradient methods. In this paper, we present a frequency domain analysis framework that interprets the momentum method as a time-variant filter for gradients, where adjustments to momentum coefficients modify the filter characteristics. Our experiments support this perspective and provide a deeper understanding of the mechanism involved. Moreover, our analysis reveals the following significant findings: high-frequency gradient components are undesired in the late stages of training; preserving the original gradient in the early stages, and gradually amplifying low-frequency gradient components during training both enhance performance. Based on these insights, we propose Frequency Stochastic Gradient Descent with Momentum (FSGDM), a heuristic optimizer that dynamically adjusts the momentum filtering characteristic with an empirically effective dynamic magnitude response. Experimental results demonstrate the superiority of FSGDM over conventional momentum optimizers. |
| title | On the Performance Analysis of Momentum Method: A Frequency Domain Perspective |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2411.19671 |