Language Model Embeddings Can Be Sufficient for Bayesian Optimization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908584545091584 |
|---|---|
| author | Nguyen, Tung Zhang, Qiuyi Yang, Bangding Lee, Chansoo Bornschein, Jorg Miao, Yingjie Perel, Sagi Chen, Yutian Song, Xingyou |
| author_facet | Nguyen, Tung Zhang, Qiuyi Yang, Bangding Lee, Chansoo Bornschein, Jorg Miao, Yingjie Perel, Sagi Chen, Yutian Song, Xingyou |
| contents | Bayesian Optimization is ubiquitous in experimental design and black-box optimization for improving search efficiency. However, most existing approaches rely on regression models which are limited to fixed search spaces and structured, tabular input features. This paper explores the use of LLM embeddings over string inputs for in-context regression in Bayesian Optimization. Our results show that representing inputs as strings enables general-purpose regression across diverse domains, including synthetic, combinatorial, and hyperparameter optimization. Furthermore, our approach achieves optimization performance comparable to state-of-the-art Gaussian Process-based methods such as Google Vizier, and demonstrates potential for broader and more flexible applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_10190 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Language Model Embeddings Can Be Sufficient for Bayesian Optimization Nguyen, Tung Zhang, Qiuyi Yang, Bangding Lee, Chansoo Bornschein, Jorg Miao, Yingjie Perel, Sagi Chen, Yutian Song, Xingyou Machine Learning Artificial Intelligence Bayesian Optimization is ubiquitous in experimental design and black-box optimization for improving search efficiency. However, most existing approaches rely on regression models which are limited to fixed search spaces and structured, tabular input features. This paper explores the use of LLM embeddings over string inputs for in-context regression in Bayesian Optimization. Our results show that representing inputs as strings enables general-purpose regression across diverse domains, including synthetic, combinatorial, and hyperparameter optimization. Furthermore, our approach achieves optimization performance comparable to state-of-the-art Gaussian Process-based methods such as Google Vizier, and demonstrates potential for broader and more flexible applications. |
| title | Language Model Embeddings Can Be Sufficient for Bayesian Optimization |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2410.10190 |