Lower-level Duality Based Reformulation and Majorization Minimization Algorithm for Hyperparameter Optimization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911787130028032 |
|---|---|
| author | Chen, He Xu, Haochen Jiang, Rujun So, Anthony Man-Cho |
| author_facet | Chen, He Xu, Haochen Jiang, Rujun So, Anthony Man-Cho |
| contents | Hyperparameter tuning is an important task of machine learning, which can be formulated as a bilevel program (BLP). However, most existing algorithms are not applicable for BLP with non-smooth lower-level problems. To address this, we propose a single-level reformulation of the BLP based on lower-level duality without involving any implicit value function. To solve the reformulation, we propose a majorization minimization algorithm that marjorizes the constraint in each iteration. Furthermore, we show that the subproblems of the proposed algorithm for several widely used hyperparameter turning models can be reformulated into conic programs that can be efficiently solved by the off-the-shelf solvers. We theoretically prove the convergence of the proposed algorithm and demonstrate its superiority through numerical experiments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_00314 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Lower-level Duality Based Reformulation and Majorization Minimization Algorithm for Hyperparameter Optimization Chen, He Xu, Haochen Jiang, Rujun So, Anthony Man-Cho Optimization and Control Hyperparameter tuning is an important task of machine learning, which can be formulated as a bilevel program (BLP). However, most existing algorithms are not applicable for BLP with non-smooth lower-level problems. To address this, we propose a single-level reformulation of the BLP based on lower-level duality without involving any implicit value function. To solve the reformulation, we propose a majorization minimization algorithm that marjorizes the constraint in each iteration. Furthermore, we show that the subproblems of the proposed algorithm for several widely used hyperparameter turning models can be reformulated into conic programs that can be efficiently solved by the off-the-shelf solvers. We theoretically prove the convergence of the proposed algorithm and demonstrate its superiority through numerical experiments. |
| title | Lower-level Duality Based Reformulation and Majorization Minimization Algorithm for Hyperparameter Optimization |
| topic | Optimization and Control |
| url | https://arxiv.org/abs/2403.00314 |