Simplifying Bayesian Optimization Via In-Context Direct Optimum Sampling
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912403654967296 |
|---|---|
| author | de Carvalho, Gustavo Sutter Pessurno Abdulrahman, Mohammed Wang, Hao Subramanian, Sriram Ganapathi St-Aubin, Marc O'Sullivan, Sharon Wan, Lawrence Ricardez-Sandoval, Luis Poupart, Pascal Kristiadi, Agustinus |
| author_facet | de Carvalho, Gustavo Sutter Pessurno Abdulrahman, Mohammed Wang, Hao Subramanian, Sriram Ganapathi St-Aubin, Marc O'Sullivan, Sharon Wan, Lawrence Ricardez-Sandoval, Luis Poupart, Pascal Kristiadi, Agustinus |
| contents | The optimization of expensive black-box functions is ubiquitous in science and engineering. A common solution to this problem is Bayesian optimization (BO), which is generally comprised of two components: (i) a surrogate model and (ii) an acquisition function, which generally require expensive re-training and optimization steps at each iteration, respectively. Although recent work enabled in-context surrogate models that do not require re-training, virtually all existing BO methods still require acquisition function maximization to select the next observation, which introduces many knobs to tune, such as Monte Carlo samplers and multi-start optimizers. In this work, we propose a completely in-context, zero-shot solution for BO that does not require surrogate fitting or acquisition function optimization. This is done by using a pre-trained deep generative model to directly sample from the posterior over the optimum point. We show that this process is equivalent to Thompson sampling and demonstrate the capabilities and cost-effectiveness of our foundation model on a suite of real-world benchmarks. We achieve an efficiency gain of more than 35x in terms of wall-clock time when compared with Gaussian process-based BO, enabling efficient parallel and distributed BO, e.g., for high-throughput optimization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_23913 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Simplifying Bayesian Optimization Via In-Context Direct Optimum Sampling de Carvalho, Gustavo Sutter Pessurno Abdulrahman, Mohammed Wang, Hao Subramanian, Sriram Ganapathi St-Aubin, Marc O'Sullivan, Sharon Wan, Lawrence Ricardez-Sandoval, Luis Poupart, Pascal Kristiadi, Agustinus Machine Learning The optimization of expensive black-box functions is ubiquitous in science and engineering. A common solution to this problem is Bayesian optimization (BO), which is generally comprised of two components: (i) a surrogate model and (ii) an acquisition function, which generally require expensive re-training and optimization steps at each iteration, respectively. Although recent work enabled in-context surrogate models that do not require re-training, virtually all existing BO methods still require acquisition function maximization to select the next observation, which introduces many knobs to tune, such as Monte Carlo samplers and multi-start optimizers. In this work, we propose a completely in-context, zero-shot solution for BO that does not require surrogate fitting or acquisition function optimization. This is done by using a pre-trained deep generative model to directly sample from the posterior over the optimum point. We show that this process is equivalent to Thompson sampling and demonstrate the capabilities and cost-effectiveness of our foundation model on a suite of real-world benchmarks. We achieve an efficiency gain of more than 35x in terms of wall-clock time when compared with Gaussian process-based BO, enabling efficient parallel and distributed BO, e.g., for high-throughput optimization. |
| title | Simplifying Bayesian Optimization Via In-Context Direct Optimum Sampling |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2505.23913 |