Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.08360 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913887420416000 |
|---|---|
| author | Seung, Hyunseok Lee, Jaewoo Ko, Hyunsuk |
| author_facet | Seung, Hyunseok Lee, Jaewoo Ko, Hyunsuk |
| contents | Adaptive gradient methods are computationally efficient and converge quickly, but they often suffer from poor generalization. In contrast, second-order methods enhance convergence and generalization but typically incur high computational and memory costs. In this work, we introduce NysAct, a scalable first-order gradient preconditioning method that strikes a balance between state-of-the-art first-order and second-order optimization methods. NysAct leverages an eigenvalue-shifted Nystrom method to approximate the activation covariance matrix, which is used as a preconditioning matrix, significantly reducing time and memory complexities with minimal impact on test accuracy. Our experiments show that NysAct not only achieves improved test accuracy compared to both first-order and second-order methods but also demands considerably less computational resources than existing second-order methods. Code is available at https://github.com/hseung88/nysact. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_08360 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | NysAct: A Scalable Preconditioned Gradient Descent using Nystrom Approximation Seung, Hyunseok Lee, Jaewoo Ko, Hyunsuk Machine Learning Adaptive gradient methods are computationally efficient and converge quickly, but they often suffer from poor generalization. In contrast, second-order methods enhance convergence and generalization but typically incur high computational and memory costs. In this work, we introduce NysAct, a scalable first-order gradient preconditioning method that strikes a balance between state-of-the-art first-order and second-order optimization methods. NysAct leverages an eigenvalue-shifted Nystrom method to approximate the activation covariance matrix, which is used as a preconditioning matrix, significantly reducing time and memory complexities with minimal impact on test accuracy. Our experiments show that NysAct not only achieves improved test accuracy compared to both first-order and second-order methods but also demands considerably less computational resources than existing second-order methods. Code is available at https://github.com/hseung88/nysact. |
| title | NysAct: A Scalable Preconditioned Gradient Descent using Nystrom Approximation |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2506.08360 |