Provably Efficient Bayesian Optimization with Unknown Gaussian Process Hyperparameter Estimation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911906799812608 |
|---|---|
| author | Ha, Huong Nguyen, Vu Tran-The, Hung Zhang, Hongyu Zhang, Xiuzhen Hengel, Anton van den |
| author_facet | Ha, Huong Nguyen, Vu Tran-The, Hung Zhang, Hongyu Zhang, Xiuzhen Hengel, Anton van den |
| contents | Gaussian process (GP) based Bayesian optimization (BO) is a powerful method for optimizing black-box functions efficiently. The practical performance and theoretical guarantees of this approach depend on having the correct GP hyperparameter values, which are usually unknown in advance and need to be estimated from the observed data. However, in practice, these estimations could be incorrect due to biased data sampling strategies used in BO. This can lead to degraded performance and break the sub-linear global convergence guarantee of BO. To address this issue, we propose a new BO method that can sub-linearly converge to the objective function's global optimum even when the true GP hyperparameters are unknown in advance and need to be estimated from the observed data. Our method uses a multi-armed bandit technique (EXP3) to add random data points to the BO process, and employs a novel training loss function for the GP hyperparameter estimation process that ensures consistent estimation. We further provide theoretical analysis of our proposed method. Finally, we demonstrate empirically that our method outperforms existing approaches on various synthetic and real-world problems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2306_06844 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Provably Efficient Bayesian Optimization with Unknown Gaussian Process Hyperparameter Estimation Ha, Huong Nguyen, Vu Tran-The, Hung Zhang, Hongyu Zhang, Xiuzhen Hengel, Anton van den Machine Learning Gaussian process (GP) based Bayesian optimization (BO) is a powerful method for optimizing black-box functions efficiently. The practical performance and theoretical guarantees of this approach depend on having the correct GP hyperparameter values, which are usually unknown in advance and need to be estimated from the observed data. However, in practice, these estimations could be incorrect due to biased data sampling strategies used in BO. This can lead to degraded performance and break the sub-linear global convergence guarantee of BO. To address this issue, we propose a new BO method that can sub-linearly converge to the objective function's global optimum even when the true GP hyperparameters are unknown in advance and need to be estimated from the observed data. Our method uses a multi-armed bandit technique (EXP3) to add random data points to the BO process, and employs a novel training loss function for the GP hyperparameter estimation process that ensures consistent estimation. We further provide theoretical analysis of our proposed method. Finally, we demonstrate empirically that our method outperforms existing approaches on various synthetic and real-world problems. |
| title | Provably Efficient Bayesian Optimization with Unknown Gaussian Process Hyperparameter Estimation |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2306.06844 |