Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.06738 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909486157922304 |
|---|---|
| author | Ivanov, Igor Volkov, Dmitrii |
| author_facet | Ivanov, Igor Volkov, Dmitrii |
| contents | Recent work showed that small changes in benchmark questions can reduce LLMs' reasoning and recall. We explore two such changes: pairing questions and adding more answer options, on three benchmarks: WMDP-bio, GPQA, and MMLU variants. We find that for more capable models, these predictably reduce performance, essentially heightening the performance ceiling of a benchmark and unsaturating it again. We suggest this approach can resurrect old benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_06738 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Resurrecting saturated LLM benchmarks with adversarial encoding Ivanov, Igor Volkov, Dmitrii Machine Learning Recent work showed that small changes in benchmark questions can reduce LLMs' reasoning and recall. We explore two such changes: pairing questions and adding more answer options, on three benchmarks: WMDP-bio, GPQA, and MMLU variants. We find that for more capable models, these predictably reduce performance, essentially heightening the performance ceiling of a benchmark and unsaturating it again. We suggest this approach can resurrect old benchmarks. |
| title | Resurrecting saturated LLM benchmarks with adversarial encoding |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2502.06738 |