Global Optimization of Gaussian Process Acquisition Functions Using a Piecewise-Linear Kernel Approximation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911609149980672 |
|---|---|
| author | Xie, Yilin Zhang, Shiqiang Paulson, Joel A. Tsay, Calvin |
| author_facet | Xie, Yilin Zhang, Shiqiang Paulson, Joel A. Tsay, Calvin |
| contents | Bayesian optimization relies on iteratively constructing and optimizing an acquisition function. The latter turns out to be a challenging, non-convex optimization problem itself. Despite the relative importance of this step, most algorithms employ sampling- or gradient-based methods, which do not provably converge to global optima. This work investigates mixed-integer programming (MIP) as a paradigm for global acquisition function optimization. Specifically, our Piecewise-linear Kernel Mixed Integer Quadratic Programming (PK-MIQP) formulation introduces a piecewise-linear approximation for Gaussian process kernels and admits a corresponding MIQP representation for acquisition functions. The proposed method is applicable to uncertainty-based acquisition functions for any stationary or dot-product kernel. We analyze the theoretical regret bounds of the proposed approximation, and empirically demonstrate the framework on synthetic functions, constrained benchmarks, and a hyperparameter tuning task. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_16893 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Global Optimization of Gaussian Process Acquisition Functions Using a Piecewise-Linear Kernel Approximation Xie, Yilin Zhang, Shiqiang Paulson, Joel A. Tsay, Calvin Optimization and Control Machine Learning Bayesian optimization relies on iteratively constructing and optimizing an acquisition function. The latter turns out to be a challenging, non-convex optimization problem itself. Despite the relative importance of this step, most algorithms employ sampling- or gradient-based methods, which do not provably converge to global optima. This work investigates mixed-integer programming (MIP) as a paradigm for global acquisition function optimization. Specifically, our Piecewise-linear Kernel Mixed Integer Quadratic Programming (PK-MIQP) formulation introduces a piecewise-linear approximation for Gaussian process kernels and admits a corresponding MIQP representation for acquisition functions. The proposed method is applicable to uncertainty-based acquisition functions for any stationary or dot-product kernel. We analyze the theoretical regret bounds of the proposed approximation, and empirically demonstrate the framework on synthetic functions, constrained benchmarks, and a hyperparameter tuning task. |
| title | Global Optimization of Gaussian Process Acquisition Functions Using a Piecewise-Linear Kernel Approximation |
| topic | Optimization and Control Machine Learning |
| url | https://arxiv.org/abs/2410.16893 |