SPRIG: Improving Large Language Model Performance by System Prompt Optimization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911565592133632 |
|---|---|
| author | Zhang, Lechen Ergen, Tolga Logeswaran, Lajanugen Lee, Moontae Jurgens, David |
| author_facet | Zhang, Lechen Ergen, Tolga Logeswaran, Lajanugen Lee, Moontae Jurgens, David |
| contents | Large Language Models (LLMs) have shown impressive capabilities in many scenarios, but their performance depends, in part, on the choice of prompt. Past research has focused on optimizing prompts specific to a task. However, much less attention has been given to optimizing the general instructions included in a prompt, known as a system prompt. To address this gap, we propose SPRIG, an edit-based genetic algorithm that iteratively constructs prompts from prespecified components to maximize the model's performance in general scenarios. We evaluate the performance of system prompts on a collection of 47 different types of tasks to ensure generalizability. Our study finds that a single optimized system prompt performs on par with task prompts optimized for each individual task. Moreover, combining system and task-level optimizations leads to further improvement, which showcases their complementary nature. Experiments also reveal that the optimized system prompts generalize effectively across model families, parameter sizes, and languages. This study provides insights into the role of system-level instructions in maximizing LLM potential. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_14826 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SPRIG: Improving Large Language Model Performance by System Prompt Optimization Zhang, Lechen Ergen, Tolga Logeswaran, Lajanugen Lee, Moontae Jurgens, David Computation and Language Artificial Intelligence Human-Computer Interaction Machine Learning Large Language Models (LLMs) have shown impressive capabilities in many scenarios, but their performance depends, in part, on the choice of prompt. Past research has focused on optimizing prompts specific to a task. However, much less attention has been given to optimizing the general instructions included in a prompt, known as a system prompt. To address this gap, we propose SPRIG, an edit-based genetic algorithm that iteratively constructs prompts from prespecified components to maximize the model's performance in general scenarios. We evaluate the performance of system prompts on a collection of 47 different types of tasks to ensure generalizability. Our study finds that a single optimized system prompt performs on par with task prompts optimized for each individual task. Moreover, combining system and task-level optimizations leads to further improvement, which showcases their complementary nature. Experiments also reveal that the optimized system prompts generalize effectively across model families, parameter sizes, and languages. This study provides insights into the role of system-level instructions in maximizing LLM potential. |
| title | SPRIG: Improving Large Language Model Performance by System Prompt Optimization |
| topic | Computation and Language Artificial Intelligence Human-Computer Interaction Machine Learning |
| url | https://arxiv.org/abs/2410.14826 |