Alternating Stochastic Variance-Reduced Algorithms with Optimal Complexity for Bilevel Optimization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917957401051136 |
|---|---|
| author | Huo, Haimei Su, Zhixun |
| author_facet | Huo, Haimei Su, Zhixun |
| contents | This paper studies the unconstrained nonconvex-strongly-convex bilevel optimization problem. A common approach to solving this problem is to alternately update the upper-level and lower-level variables using (biased) stochastic gradients or their variants, with the lower-level variable updated either one step or multiple steps. In this context, we propose two alternating stochastic variance-reduced algorithms, namely ALS-SPIDER and ALS-STORM, which introduce an auxiliary variable to estimate the hypergradient for updating the upper-level variable. ALS-SPIDER employs the SPIDER estimator for updating variables, while ALS-STORM is a modification of ALS-SPIDER designed to avoid using large batch sizes in every iteration. Theoretically, both algorithms can find an $ε$-stationary point of the bilevel problem with a sample complexity of $O(ε^{-1.5})$ for arbitrary constant number of lower-level variable updates. To the best of our knowledge, they are the first algorithms to achieve the optimal complexity of $O(ε^{-1.5})$ when performing multiple updates on the lower-level variable. Numerical experiments are conducted to illustrate the efficiency of our algorithms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_11377 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Alternating Stochastic Variance-Reduced Algorithms with Optimal Complexity for Bilevel Optimization Huo, Haimei Su, Zhixun Optimization and Control This paper studies the unconstrained nonconvex-strongly-convex bilevel optimization problem. A common approach to solving this problem is to alternately update the upper-level and lower-level variables using (biased) stochastic gradients or their variants, with the lower-level variable updated either one step or multiple steps. In this context, we propose two alternating stochastic variance-reduced algorithms, namely ALS-SPIDER and ALS-STORM, which introduce an auxiliary variable to estimate the hypergradient for updating the upper-level variable. ALS-SPIDER employs the SPIDER estimator for updating variables, while ALS-STORM is a modification of ALS-SPIDER designed to avoid using large batch sizes in every iteration. Theoretically, both algorithms can find an $ε$-stationary point of the bilevel problem with a sample complexity of $O(ε^{-1.5})$ for arbitrary constant number of lower-level variable updates. To the best of our knowledge, they are the first algorithms to achieve the optimal complexity of $O(ε^{-1.5})$ when performing multiple updates on the lower-level variable. Numerical experiments are conducted to illustrate the efficiency of our algorithms. |
| title | Alternating Stochastic Variance-Reduced Algorithms with Optimal Complexity for Bilevel Optimization |
| topic | Optimization and Control |
| url | https://arxiv.org/abs/2404.11377 |