Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.07820 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909633988263936 |
|---|---|
| author | Schneider, Lennart Wistuba, Martin Klein, Aaron Golebiowski, Jacek Zappella, Giovanni Merra, Felice Antonio |
| author_facet | Schneider, Lennart Wistuba, Martin Klein, Aaron Golebiowski, Jacek Zappella, Giovanni Merra, Felice Antonio |
| contents | Optimal prompt selection is crucial for maximizing large language model (LLM) performance on downstream tasks, especially in black-box settings where models are only accessible via APIs. Black-box prompt selection is challenging due to potentially large, combinatorial search spaces, absence of gradient information, and high evaluation cost of prompts on a validation set. We propose HbBoPs, a novel method that combines a structural-aware deep kernel Gaussian Process with Hyperband as a multi-fidelity scheduler to efficiently select prompts. HbBoPs uses embeddings of instructions and few-shot exemplars, treating them as modular components within prompts. This enhances the surrogate model's ability to predict which prompt to evaluate next in a sample-efficient manner. Hyperband improves query-efficiency by adaptively allocating resources across different fidelity levels, reducing the number of validation instances required for evaluating prompts. Extensive experiments across ten diverse benchmarks and three LLMs demonstrate that HbBoPs outperforms state-of-the-art methods in both performance and efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_07820 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Hyperband-based Bayesian Optimization for Black-box Prompt Selection Schneider, Lennart Wistuba, Martin Klein, Aaron Golebiowski, Jacek Zappella, Giovanni Merra, Felice Antonio Machine Learning Artificial Intelligence Optimal prompt selection is crucial for maximizing large language model (LLM) performance on downstream tasks, especially in black-box settings where models are only accessible via APIs. Black-box prompt selection is challenging due to potentially large, combinatorial search spaces, absence of gradient information, and high evaluation cost of prompts on a validation set. We propose HbBoPs, a novel method that combines a structural-aware deep kernel Gaussian Process with Hyperband as a multi-fidelity scheduler to efficiently select prompts. HbBoPs uses embeddings of instructions and few-shot exemplars, treating them as modular components within prompts. This enhances the surrogate model's ability to predict which prompt to evaluate next in a sample-efficient manner. Hyperband improves query-efficiency by adaptively allocating resources across different fidelity levels, reducing the number of validation instances required for evaluating prompts. Extensive experiments across ten diverse benchmarks and three LLMs demonstrate that HbBoPs outperforms state-of-the-art methods in both performance and efficiency. |
| title | Hyperband-based Bayesian Optimization for Black-box Prompt Selection |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2412.07820 |