Examining the robustness of LLM evaluation to the distributional assumptions of benchmarks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909218044379136 |
|---|---|
| author | Ailem, Melissa Marazopoulou, Katerina Siska, Charlotte Bono, James |
| author_facet | Ailem, Melissa Marazopoulou, Katerina Siska, Charlotte Bono, James |
| contents | Benchmarks have emerged as the central approach for evaluating Large Language Models (LLMs). The research community often relies on a model's average performance across the test prompts of a benchmark to evaluate the model's performance. This is consistent with the assumption that the test prompts within a benchmark represent a random sample from a real-world distribution of interest. We note that this is generally not the case; instead, we hold that the distribution of interest varies according to the specific use case. We find that (1) the correlation in model performance across test prompts is non-random, (2) accounting for correlations across test prompts can change model rankings on major benchmarks, (3) explanatory factors for these correlations include semantic similarity and common LLM failure points. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_16966 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Examining the robustness of LLM evaluation to the distributional assumptions of benchmarks Ailem, Melissa Marazopoulou, Katerina Siska, Charlotte Bono, James Computation and Language Benchmarks have emerged as the central approach for evaluating Large Language Models (LLMs). The research community often relies on a model's average performance across the test prompts of a benchmark to evaluate the model's performance. This is consistent with the assumption that the test prompts within a benchmark represent a random sample from a real-world distribution of interest. We note that this is generally not the case; instead, we hold that the distribution of interest varies according to the specific use case. We find that (1) the correlation in model performance across test prompts is non-random, (2) accounting for correlations across test prompts can change model rankings on major benchmarks, (3) explanatory factors for these correlations include semantic similarity and common LLM failure points. |
| title | Examining the robustness of LLM evaluation to the distributional assumptions of benchmarks |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2404.16966 |