State of What Art? A Call for Multi-Prompt LLM Evaluation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917657811353600 |
|---|---|
| author | Mizrahi, Moran Kaplan, Guy Malkin, Dan Dror, Rotem Shahaf, Dafna Stanovsky, Gabriel |
| author_facet | Mizrahi, Moran Kaplan, Guy Malkin, Dan Dror, Rotem Shahaf, Dafna Stanovsky, Gabriel |
| contents | Recent advances in large language models (LLMs) have led to the development of various evaluation benchmarks. These benchmarks typically rely on a single instruction template for evaluating all LLMs on a specific task. In this paper, we comprehensively analyze the brittleness of results obtained via single-prompt evaluations across 6.5M instances, involving 20 different LLMs and 39 tasks from 3 benchmarks. To improve robustness of the analysis, we propose to evaluate LLMs with a set of diverse prompts instead. We discuss tailored evaluation metrics for specific use cases (e.g., LLM developers vs. developers interested in a specific downstream task), ensuring a more reliable and meaningful assessment of LLM capabilities. We then implement these criteria and conduct evaluations of multiple models, providing insights into the true strengths and limitations of current LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_00595 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | State of What Art? A Call for Multi-Prompt LLM Evaluation Mizrahi, Moran Kaplan, Guy Malkin, Dan Dror, Rotem Shahaf, Dafna Stanovsky, Gabriel Computation and Language Recent advances in large language models (LLMs) have led to the development of various evaluation benchmarks. These benchmarks typically rely on a single instruction template for evaluating all LLMs on a specific task. In this paper, we comprehensively analyze the brittleness of results obtained via single-prompt evaluations across 6.5M instances, involving 20 different LLMs and 39 tasks from 3 benchmarks. To improve robustness of the analysis, we propose to evaluate LLMs with a set of diverse prompts instead. We discuss tailored evaluation metrics for specific use cases (e.g., LLM developers vs. developers interested in a specific downstream task), ensuring a more reliable and meaningful assessment of LLM capabilities. We then implement these criteria and conduct evaluations of multiple models, providing insights into the true strengths and limitations of current LLMs. |
| title | State of What Art? A Call for Multi-Prompt LLM Evaluation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2401.00595 |