Does Prompt Formatting Have Any Impact on LLM Performance?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917839334539264 |
|---|---|
| author | He, Jia Rungta, Mukund Koleczek, David Sekhon, Arshdeep Wang, Franklin X Hasan, Sadid |
| author_facet | He, Jia Rungta, Mukund Koleczek, David Sekhon, Arshdeep Wang, Franklin X Hasan, Sadid |
| contents | In the realm of Large Language Models (LLMs), prompt optimization is crucial for model performance. Although previous research has explored aspects like rephrasing prompt contexts, using various prompting techniques (like in-context learning and chain-of-thought), and ordering few-shot examples, our understanding of LLM sensitivity to prompt templates remains limited. Therefore, this paper examines the impact of different prompt templates on LLM performance. We formatted the same contexts into various human-readable templates, including plain text, Markdown, JSON, and YAML, and evaluated their impact across tasks like natural language reasoning, code generation, and translation using OpenAI's GPT models. Experiments show that GPT-3.5-turbo's performance varies by up to 40\% in a code translation task depending on the prompt template, while larger models like GPT-4 are more robust to these variations. Our analysis highlights the need to reconsider the use of fixed prompt templates, as different formats can significantly affect model performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_10541 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Does Prompt Formatting Have Any Impact on LLM Performance? He, Jia Rungta, Mukund Koleczek, David Sekhon, Arshdeep Wang, Franklin X Hasan, Sadid Computation and Language Machine Learning In the realm of Large Language Models (LLMs), prompt optimization is crucial for model performance. Although previous research has explored aspects like rephrasing prompt contexts, using various prompting techniques (like in-context learning and chain-of-thought), and ordering few-shot examples, our understanding of LLM sensitivity to prompt templates remains limited. Therefore, this paper examines the impact of different prompt templates on LLM performance. We formatted the same contexts into various human-readable templates, including plain text, Markdown, JSON, and YAML, and evaluated their impact across tasks like natural language reasoning, code generation, and translation using OpenAI's GPT models. Experiments show that GPT-3.5-turbo's performance varies by up to 40\% in a code translation task depending on the prompt template, while larger models like GPT-4 are more robust to these variations. Our analysis highlights the need to reconsider the use of fixed prompt templates, as different formats can significantly affect model performance. |
| title | Does Prompt Formatting Have Any Impact on LLM Performance? |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2411.10541 |