Decentralized projected Riemannian stochastic recursive momentum method for nonconvex optimization
Fuente:
arXiv
Guardado en:
| Autores principales: | , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915244682510336 |
|---|---|
| author | Deng, Kangkang Hu, Jiang |
| author_facet | Deng, Kangkang Hu, Jiang |
| contents | This paper studies decentralized optimization over a compact submanifold within a communication network of $n$ nodes, where each node possesses a smooth non-convex local cost function, and the goal is to jointly minimize the sum of these local costs. We focus particularly on the online setting, where local data is processed in real-time as it streams in, without the need for full data storage. We propose a decentralized projected Riemannian stochastic recursive momentum (DPRSRM) method that employs local hybrid stochastic gradient estimators and uses the network to track the global gradient. DPRSRM achieves an oracle complexity of \(\mathcal{O}(ε^{-\frac{3}{2}})\), outperforming existing methods that have at most \(\mathcal{O}(ε^{-2})\) complexity. Our method requires only $\mathcal{O}(1)$ gradient evaluations per iteration for each local node and does not require restarting with a large batch gradient. Furthermore, we demonstrate the effectiveness of our proposed methods compared to state-of-the-art ones through numerical experiments on principal component analysis problems and low-rank matrix completion. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_02382 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Decentralized projected Riemannian stochastic recursive momentum method for nonconvex optimization Deng, Kangkang Hu, Jiang Optimization and Control 90C06, 90C22, 90C26, 90C56 This paper studies decentralized optimization over a compact submanifold within a communication network of $n$ nodes, where each node possesses a smooth non-convex local cost function, and the goal is to jointly minimize the sum of these local costs. We focus particularly on the online setting, where local data is processed in real-time as it streams in, without the need for full data storage. We propose a decentralized projected Riemannian stochastic recursive momentum (DPRSRM) method that employs local hybrid stochastic gradient estimators and uses the network to track the global gradient. DPRSRM achieves an oracle complexity of \(\mathcal{O}(ε^{-\frac{3}{2}})\), outperforming existing methods that have at most \(\mathcal{O}(ε^{-2})\) complexity. Our method requires only $\mathcal{O}(1)$ gradient evaluations per iteration for each local node and does not require restarting with a large batch gradient. Furthermore, we demonstrate the effectiveness of our proposed methods compared to state-of-the-art ones through numerical experiments on principal component analysis problems and low-rank matrix completion. |
| title | Decentralized projected Riemannian stochastic recursive momentum method for nonconvex optimization |
| topic | Optimization and Control 90C06, 90C22, 90C26, 90C56 |
| url | https://arxiv.org/abs/2412.02382 |