Convergence Rate Analysis of SOAP with Arbitrary Orthogonal Projection Matrices
Fuente:
arXiv
Salvato in:
| Autori principali: | , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908993126924288 |
|---|---|
| author | Li, Huan Lin, Zhouchen |
| author_facet | Li, Huan Lin, Zhouchen |
| contents | In this short note, we establish, for the first time, the convergence rate of SOAP, an efficient and popular matrix-based optimizer for training deep neural networks. Our analysis extends to a more general variant of SOAP that admits arbitrary orthogonal projection matrices and requires only that these matrices be conditionally independent of the current stochastic gradient at each iteration. For example, they may be constructed from information available up to the preceding step. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_21616 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Convergence Rate Analysis of SOAP with Arbitrary Orthogonal Projection Matrices Li, Huan Lin, Zhouchen Optimization and Control In this short note, we establish, for the first time, the convergence rate of SOAP, an efficient and popular matrix-based optimizer for training deep neural networks. Our analysis extends to a more general variant of SOAP that admits arbitrary orthogonal projection matrices and requires only that these matrices be conditionally independent of the current stochastic gradient at each iteration. For example, they may be constructed from information available up to the preceding step. |
| title | Convergence Rate Analysis of SOAP with Arbitrary Orthogonal Projection Matrices |
| topic | Optimization and Control |
| url | https://arxiv.org/abs/2604.21616 |