Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.11634 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913523044450304 |
|---|---|
| author | Moore, Kyle Roberts, Jesse Pham, Thao Ewaleifoh, Oseremhen Fisher, Doug |
| author_facet | Moore, Kyle Roberts, Jesse Pham, Thao Ewaleifoh, Oseremhen Fisher, Doug |
| contents | Cloze testing is a common method for measuring the behavior of large language models on a number of benchmark tasks. Using the MMLU dataset, we show that the base-rate probability (BRP) differences across answer tokens are significant and affect task performance ie. guess A if uncertain. We find that counterfactual prompting does sufficiently mitigate the BRP effect. The BRP effect is found to have a similar effect to test taking strategies employed by humans leading to the conflation of task performance and test-taking ability. We propose the Nvr-X-MMLU task, a variation of MMLU, which helps to disambiguate test-taking ability from task performance and reports the latter. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_11634 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Base-Rate Effect on LLM Benchmark Performance: Disambiguating Test-Taking Strategies from Benchmark Performance Moore, Kyle Roberts, Jesse Pham, Thao Ewaleifoh, Oseremhen Fisher, Doug Computation and Language Artificial Intelligence Cloze testing is a common method for measuring the behavior of large language models on a number of benchmark tasks. Using the MMLU dataset, we show that the base-rate probability (BRP) differences across answer tokens are significant and affect task performance ie. guess A if uncertain. We find that counterfactual prompting does sufficiently mitigate the BRP effect. The BRP effect is found to have a similar effect to test taking strategies employed by humans leading to the conflation of task performance and test-taking ability. We propose the Nvr-X-MMLU task, a variation of MMLU, which helps to disambiguate test-taking ability from task performance and reports the latter. |
| title | The Base-Rate Effect on LLM Benchmark Performance: Disambiguating Test-Taking Strategies from Benchmark Performance |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2406.11634 |