Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.16763 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911735471931392 |
|---|---|
| author | Akhtar, Mubashara Reuel, Anka Soni, Prajna Ahuja, Sanchit Ammanamanchi, Pawan Sasanka Rawal, Ruchit Zouhar, Vilém Yadav, Srishti Whitehouse, Chenxi Ki, Dayeon Mickel, Jennifer Choshen, Leshem Šuppa, Marek Batzner, Jan Chim, Jenny Sania, Jeba Long, Yanan Rahmani, Hossein A. Knight, Christina Nan, Yiyang Raj, Jyoutir Fan, Yu Singh, Shubham Sahoo, Subramanyam Habba, Eliya Gohar, Usman Pawar, Siddhesh Scholz, Robert Subramonian, Arjun Ni, Jingwei Kochenderfer, Mykel Koyejo, Sanmi Sachan, Mrinmaya Biderman, Stella Talat, Zeerak Ghosh, Avijit Solaiman, Irene |
| author_facet | Akhtar, Mubashara Reuel, Anka Soni, Prajna Ahuja, Sanchit Ammanamanchi, Pawan Sasanka Rawal, Ruchit Zouhar, Vilém Yadav, Srishti Whitehouse, Chenxi Ki, Dayeon Mickel, Jennifer Choshen, Leshem Šuppa, Marek Batzner, Jan Chim, Jenny Sania, Jeba Long, Yanan Rahmani, Hossein A. Knight, Christina Nan, Yiyang Raj, Jyoutir Fan, Yu Singh, Shubham Sahoo, Subramanyam Habba, Eliya Gohar, Usman Pawar, Siddhesh Scholz, Robert Subramonian, Arjun Ni, Jingwei Kochenderfer, Mykel Koyejo, Sanmi Sachan, Mrinmaya Biderman, Stella Talat, Zeerak Ghosh, Avijit Solaiman, Irene |
| contents | Artificial intelligence benchmarks are an important mechanism for measuring model progress and guiding deployment decisions. However, benchmarks quickly "saturate", making it difficult to differentiate models and diminishing their long-term value. In this study, we define benchmark saturation and analyze it across 60 language model benchmarks using 14 properties that relate to saturation. We find that nearly half of the our benchmarks exhibit saturation, with rates increasing with age. Further, we find that resilience to saturation is impacted by expert-curation, not by public test data. Our results suggest that design choices can extend benchmark longevity and inform more durable evaluation approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_16763 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation Akhtar, Mubashara Reuel, Anka Soni, Prajna Ahuja, Sanchit Ammanamanchi, Pawan Sasanka Rawal, Ruchit Zouhar, Vilém Yadav, Srishti Whitehouse, Chenxi Ki, Dayeon Mickel, Jennifer Choshen, Leshem Šuppa, Marek Batzner, Jan Chim, Jenny Sania, Jeba Long, Yanan Rahmani, Hossein A. Knight, Christina Nan, Yiyang Raj, Jyoutir Fan, Yu Singh, Shubham Sahoo, Subramanyam Habba, Eliya Gohar, Usman Pawar, Siddhesh Scholz, Robert Subramonian, Arjun Ni, Jingwei Kochenderfer, Mykel Koyejo, Sanmi Sachan, Mrinmaya Biderman, Stella Talat, Zeerak Ghosh, Avijit Solaiman, Irene Artificial Intelligence Artificial intelligence benchmarks are an important mechanism for measuring model progress and guiding deployment decisions. However, benchmarks quickly "saturate", making it difficult to differentiate models and diminishing their long-term value. In this study, we define benchmark saturation and analyze it across 60 language model benchmarks using 14 properties that relate to saturation. We find that nearly half of the our benchmarks exhibit saturation, with rates increasing with age. Further, we find that resilience to saturation is impacted by expert-curation, not by public test data. Our results suggest that design choices can extend benchmark longevity and inform more durable evaluation approaches. |
| title | When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2602.16763 |