An Adaptively Inexact Method for Bilevel Learning Using Primal-Dual Style Differentiation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915332335075328 |
|---|---|
| author | Bogensperger, Lea Ehrhardt, Matthias J. Pock, Thomas Salehi, Mohammad Sadegh Wong, Hok Shing |
| author_facet | Bogensperger, Lea Ehrhardt, Matthias J. Pock, Thomas Salehi, Mohammad Sadegh Wong, Hok Shing |
| contents | We consider a bilevel learning framework for learning linear operators. In this framework, the learnable parameters are optimized via a loss function that also depends on the minimizer of a convex optimization problem (denoted lower-level problem). We utilize an iterative algorithm called `piggyback' to compute the gradient of the loss and minimizer of the lower-level problem. Given that the lower-level problem is solved numerically, the loss function and thus its gradient can only be computed inexactly. To estimate the accuracy of the computed hypergradient, we derive an a-posteriori error bound, which provides guides for setting the tolerance for the lower-level problem, as well as the piggyback algorithm. To efficiently solve the upper-level optimization, we also propose an adaptive method for choosing a suitable step-size. To illustrate the proposed method, we consider a few learned regularizer problems, such as training an input-convex neural network. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_06436 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | An Adaptively Inexact Method for Bilevel Learning Using Primal-Dual Style Differentiation Bogensperger, Lea Ehrhardt, Matthias J. Pock, Thomas Salehi, Mohammad Sadegh Wong, Hok Shing Optimization and Control Machine Learning We consider a bilevel learning framework for learning linear operators. In this framework, the learnable parameters are optimized via a loss function that also depends on the minimizer of a convex optimization problem (denoted lower-level problem). We utilize an iterative algorithm called `piggyback' to compute the gradient of the loss and minimizer of the lower-level problem. Given that the lower-level problem is solved numerically, the loss function and thus its gradient can only be computed inexactly. To estimate the accuracy of the computed hypergradient, we derive an a-posteriori error bound, which provides guides for setting the tolerance for the lower-level problem, as well as the piggyback algorithm. To efficiently solve the upper-level optimization, we also propose an adaptive method for choosing a suitable step-size. To illustrate the proposed method, we consider a few learned regularizer problems, such as training an input-convex neural network. |
| title | An Adaptively Inexact Method for Bilevel Learning Using Primal-Dual Style Differentiation |
| topic | Optimization and Control Machine Learning |
| url | https://arxiv.org/abs/2412.06436 |