Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.09556 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929678505213952 |
|---|---|
| author | Kopal, Jakub Gregor, Michal de Leon-Martinez, Santiago Simko, Jakub |
| author_facet | Kopal, Jakub Gregor, Michal de Leon-Martinez, Santiago Simko, Jakub |
| contents | Overshoot is a novel, momentum-based stochastic gradient descent optimization method designed to enhance performance beyond standard and Nesterov's momentum. In conventional momentum methods, gradients from previous steps are aggregated with the gradient at current model weights before taking a step and updating the model. Rather than calculating gradient at the current model weights, Overshoot calculates the gradient at model weights shifted in the direction of the current momentum. This sacrifices the immediate benefit of using the gradient w.r.t. the exact model weights now, in favor of evaluating at a point, which will likely be more relevant for future updates. We show that incorporating this principle into momentum-based optimizers (SGD with momentum and Adam) results in faster convergence (saving on average at least 15% of steps). Overshoot consistently outperforms both standard and Nesterov's momentum across a wide range of tasks and integrates into popular momentum-based optimizers with zero memory and small computational overhead. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_09556 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Overshoot: Taking advantage of future gradients in momentum-based stochastic optimization Kopal, Jakub Gregor, Michal de Leon-Martinez, Santiago Simko, Jakub Machine Learning Overshoot is a novel, momentum-based stochastic gradient descent optimization method designed to enhance performance beyond standard and Nesterov's momentum. In conventional momentum methods, gradients from previous steps are aggregated with the gradient at current model weights before taking a step and updating the model. Rather than calculating gradient at the current model weights, Overshoot calculates the gradient at model weights shifted in the direction of the current momentum. This sacrifices the immediate benefit of using the gradient w.r.t. the exact model weights now, in favor of evaluating at a point, which will likely be more relevant for future updates. We show that incorporating this principle into momentum-based optimizers (SGD with momentum and Adam) results in faster convergence (saving on average at least 15% of steps). Overshoot consistently outperforms both standard and Nesterov's momentum across a wide range of tasks and integrates into popular momentum-based optimizers with zero memory and small computational overhead. |
| title | Overshoot: Taking advantage of future gradients in momentum-based stochastic optimization |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2501.09556 |