Automatic Prompt Selection for Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929302245736448 |
|---|---|
| author | Do, Viet-Tung Hoang, Van-Khanh Nguyen, Duy-Hung Sabahi, Shahab Yang, Jeff Hotta, Hajime Nguyen, Minh-Tien Le, Hung |
| author_facet | Do, Viet-Tung Hoang, Van-Khanh Nguyen, Duy-Hung Sabahi, Shahab Yang, Jeff Hotta, Hajime Nguyen, Minh-Tien Le, Hung |
| contents | Large Language Models (LLMs) can perform various natural language processing tasks with suitable instruction prompts. However, designing effective prompts manually is challenging and time-consuming. Existing methods for automatic prompt optimization either lack flexibility or efficiency. In this paper, we propose an effective approach to automatically select the optimal prompt for a given input from a finite set of synthetic candidate prompts. Our approach consists of three steps: (1) clustering the training data and generating candidate prompts for each cluster using an LLM-based prompt generator; (2) synthesizing a dataset of input-prompt-output tuples for training a prompt evaluator to rank the prompts based on their relevance to the input; (3) using the prompt evaluator to select the best prompt for a new input at test time. Our approach balances prompt generality-specificity and eliminates the need for resource-intensive training and inference. It demonstrates competitive performance on zero-shot question-answering datasets: GSM8K, MultiArith, and AQuA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_02717 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Automatic Prompt Selection for Large Language Models Do, Viet-Tung Hoang, Van-Khanh Nguyen, Duy-Hung Sabahi, Shahab Yang, Jeff Hotta, Hajime Nguyen, Minh-Tien Le, Hung Computation and Language Machine Learning Large Language Models (LLMs) can perform various natural language processing tasks with suitable instruction prompts. However, designing effective prompts manually is challenging and time-consuming. Existing methods for automatic prompt optimization either lack flexibility or efficiency. In this paper, we propose an effective approach to automatically select the optimal prompt for a given input from a finite set of synthetic candidate prompts. Our approach consists of three steps: (1) clustering the training data and generating candidate prompts for each cluster using an LLM-based prompt generator; (2) synthesizing a dataset of input-prompt-output tuples for training a prompt evaluator to rank the prompts based on their relevance to the input; (3) using the prompt evaluator to select the best prompt for a new input at test time. Our approach balances prompt generality-specificity and eliminates the need for resource-intensive training and inference. It demonstrates competitive performance on zero-shot question-answering datasets: GSM8K, MultiArith, and AQuA. |
| title | Automatic Prompt Selection for Large Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2404.02717 |