Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleading
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914520310480896 |
|---|---|
| author | Sadjoli, Nicholas Siefken, Tim Ghosh, Atin Mai, Yifan Dahlmeier, Daniel |
| author_facet | Sadjoli, Nicholas Siefken, Tim Ghosh, Atin Mai, Yifan Dahlmeier, Daniel |
| contents | Current Large Language Model (LLM) evaluation frameworks utilize the same static prompt template across all models under evaluation. This differs from the common industry practice of using prompt optimization (PO) techniques to optimize the prompt for each model to maximize application performance. In this paper, we investigate the effect of PO towards LLM evaluations. Our results on public academic and internal industry benchmarks show that PO greatly affects the final ranking of models. This highlights the importance of practitioners performing PO per model when conducting evaluations to choose the best model for a given task. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_27637 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleading Sadjoli, Nicholas Siefken, Tim Ghosh, Atin Mai, Yifan Dahlmeier, Daniel Artificial Intelligence 68T01 Current Large Language Model (LLM) evaluation frameworks utilize the same static prompt template across all models under evaluation. This differs from the common industry practice of using prompt optimization (PO) techniques to optimize the prompt for each model to maximize application performance. In this paper, we investigate the effect of PO towards LLM evaluations. Our results on public academic and internal industry benchmarks show that PO greatly affects the final ranking of models. This highlights the importance of practitioners performing PO per model when conducting evaluations to choose the best model for a given task. |
| title | Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleading |
| topic | Artificial Intelligence 68T01 |
| url | https://arxiv.org/abs/2604.27637 |