Det-CGD: Compressed Gradient Descent with Matrix Stepsizes for Non-Convex Optimization
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929322446553088 |
|---|---|
| author | Li, Hanmin Karagulyan, Avetik Richtárik, Peter |
| author_facet | Li, Hanmin Karagulyan, Avetik Richtárik, Peter |
| contents | This paper introduces a new method for minimizing matrix-smooth non-convex objectives through the use of novel Compressed Gradient Descent (CGD) algorithms enhanced with a matrix-valued stepsize. The proposed algorithms are theoretically analyzed first in the single-node and subsequently in the distributed settings. Our theoretical results reveal that the matrix stepsize in CGD can capture the objective's structure and lead to faster convergence compared to a scalar stepsize. As a byproduct of our general results, we emphasize the importance of selecting the compression mechanism and the matrix stepsize in a layer-wise manner, taking advantage of model structure. Moreover, we provide theoretical guarantees for free compression, by designing specific layer-wise compressors for the non-convex matrix smooth objectives. Our findings are supported with empirical evidence. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_12568 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Det-CGD: Compressed Gradient Descent with Matrix Stepsizes for Non-Convex Optimization Li, Hanmin Karagulyan, Avetik Richtárik, Peter Optimization and Control 90C26 This paper introduces a new method for minimizing matrix-smooth non-convex objectives through the use of novel Compressed Gradient Descent (CGD) algorithms enhanced with a matrix-valued stepsize. The proposed algorithms are theoretically analyzed first in the single-node and subsequently in the distributed settings. Our theoretical results reveal that the matrix stepsize in CGD can capture the objective's structure and lead to faster convergence compared to a scalar stepsize. As a byproduct of our general results, we emphasize the importance of selecting the compression mechanism and the matrix stepsize in a layer-wise manner, taking advantage of model structure. Moreover, we provide theoretical guarantees for free compression, by designing specific layer-wise compressors for the non-convex matrix smooth objectives. Our findings are supported with empirical evidence. |
| title | Det-CGD: Compressed Gradient Descent with Matrix Stepsizes for Non-Convex Optimization |
| topic | Optimization and Control 90C26 |
| url | https://arxiv.org/abs/2305.12568 |