SAPD+: An Accelerated Stochastic Method for Nonconvex-Concave Minimax Problems
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2022
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910646644244480 |
|---|---|
| author | Zhang, Xuan Aybat, Necdet Serhat Gürbüzbalaban, Mert |
| author_facet | Zhang, Xuan Aybat, Necdet Serhat Gürbüzbalaban, Mert |
| contents | We propose a new stochastic method SAPD+ for solving nonconvex-concave minimax problems of the form $\min\max\mathcal{L}(x,y)=f(x)+Φ(x,y)-g(y)$, where $f,g$ are closed convex and $Φ(x,y)$ is a smooth function that is weakly convex in $x$, (strongly) concave in $y$. Let $δ^2$ denote the variance bound for the unbiased stochastic oracle used within SAPD+ to estimate $\nablaΦ$. When $δ>0$, for both strongly concave and merely concave settings, SAPD+ achieves the best known oracle complexities: $\mathcal{O}\Big(κ_y\max\Big\{1,\frac{δ^2}{ε^2}\Big\}\frac{L\mathcal{G}_0}{ε^{2}}\Big)$ for the strongly concave case without assuming compactness of the problem domain, and $\mathcal{O}\Big(\frac{L^3\mathcal{D}_y^2\mathcal{G}_0}{ε^{4}}\Big(1+\frac{δ^2}{ε^2}\Big)\Big)$ for the merely concave case, where $κ_y\geq 1$ is the condition number, $L$ is the Lipschitz constant of $\nabla Φ$, $\mathcal{G}_0$ is the primal-dual gap of the initial point, and $\mathcal{D}_y=\sup\{\|y\|:\ y\in\mathbf{dom} g\}$. We also propose SAPD+ with variance reduction, which enjoys $\mathcal{O}\Big(\max\Big\{κ_y,\sqrt{\fracδε}\Big\}\cdot (1+κ_y\fracδε)\frac{L\mathcal{G}_0}{ε^2}\Big)$ oracle complexity for weakly convex-strongly concave setting --this is the best known upper complexity bound in the literature for this setting and our paper establishes it for the first time. We demonstrate the efficiency of SAPD+ on a distributionally robust learning problem with a nonconvex regularizer and also on a multi-class classification problem in deep learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2205_15084 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | SAPD+: An Accelerated Stochastic Method for Nonconvex-Concave Minimax Problems Zhang, Xuan Aybat, Necdet Serhat Gürbüzbalaban, Mert Optimization and Control We propose a new stochastic method SAPD+ for solving nonconvex-concave minimax problems of the form $\min\max\mathcal{L}(x,y)=f(x)+Φ(x,y)-g(y)$, where $f,g$ are closed convex and $Φ(x,y)$ is a smooth function that is weakly convex in $x$, (strongly) concave in $y$. Let $δ^2$ denote the variance bound for the unbiased stochastic oracle used within SAPD+ to estimate $\nablaΦ$. When $δ>0$, for both strongly concave and merely concave settings, SAPD+ achieves the best known oracle complexities: $\mathcal{O}\Big(κ_y\max\Big\{1,\frac{δ^2}{ε^2}\Big\}\frac{L\mathcal{G}_0}{ε^{2}}\Big)$ for the strongly concave case without assuming compactness of the problem domain, and $\mathcal{O}\Big(\frac{L^3\mathcal{D}_y^2\mathcal{G}_0}{ε^{4}}\Big(1+\frac{δ^2}{ε^2}\Big)\Big)$ for the merely concave case, where $κ_y\geq 1$ is the condition number, $L$ is the Lipschitz constant of $\nabla Φ$, $\mathcal{G}_0$ is the primal-dual gap of the initial point, and $\mathcal{D}_y=\sup\{\|y\|:\ y\in\mathbf{dom} g\}$. We also propose SAPD+ with variance reduction, which enjoys $\mathcal{O}\Big(\max\Big\{κ_y,\sqrt{\fracδε}\Big\}\cdot (1+κ_y\fracδε)\frac{L\mathcal{G}_0}{ε^2}\Big)$ oracle complexity for weakly convex-strongly concave setting --this is the best known upper complexity bound in the literature for this setting and our paper establishes it for the first time. We demonstrate the efficiency of SAPD+ on a distributionally robust learning problem with a nonconvex regularizer and also on a multi-class classification problem in deep learning. |
| title | SAPD+: An Accelerated Stochastic Method for Nonconvex-Concave Minimax Problems |
| topic | Optimization and Control |
| url | https://arxiv.org/abs/2205.15084 |