Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.11588 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911316036288512 |
|---|---|
| author | von Laszewski, Gregor Brewer, Wesley Thiyagalingam, Jeyan Papay, Juri Foundjem, Armstrong Luszczek, Piotr Emani, Murali Moore, Shirley V. Reddi, Vijay Janapa Sinclair, Matthew D. Lobentanzer, Sebastian Goswami, Sujata Hawks, Benjamin Colombo, Marco Tran, Nhan Kirkpatrick, Christine R. Alsudais, Abdulkareem Barrett, Gregg Li, Tianhao Morehouse, Kirsten Venkataraman, Shivaram Jain, Rutwik Mathur, Kartik Lu, Victor Singh, Tejinder Mirza, Khojasteh Z. Chen, Kongtao Kunapuli, Sasidhar Farrell, Gavin Umeton, Renato Fox, Geoffrey C. |
| author_facet | von Laszewski, Gregor Brewer, Wesley Thiyagalingam, Jeyan Papay, Juri Foundjem, Armstrong Luszczek, Piotr Emani, Murali Moore, Shirley V. Reddi, Vijay Janapa Sinclair, Matthew D. Lobentanzer, Sebastian Goswami, Sujata Hawks, Benjamin Colombo, Marco Tran, Nhan Kirkpatrick, Christine R. Alsudais, Abdulkareem Barrett, Gregg Li, Tianhao Morehouse, Kirsten Venkataraman, Shivaram Jain, Rutwik Mathur, Kartik Lu, Victor Singh, Tejinder Mirza, Khojasteh Z. Chen, Kongtao Kunapuli, Sasidhar Farrell, Gavin Umeton, Renato Fox, Geoffrey C. |
| contents | Benchmarks are a cornerstone of modern machine learning, enabling reproducibility, comparison, and scientific progress. However, AI benchmarks are increasingly complex, requiring dynamic, AI-focused workflows. Rapid evolution in model architectures, scale, datasets, and deployment contexts makes evaluation a moving target. Large language models often memorize static benchmarks, causing a gap between benchmark results and real-world performance.
Beyond traditional static benchmarks, continuous adaptive benchmarking frameworks are needed to align scientific assessment with deployment risks. This calls for skills and education in AI Benchmark Carpentry. From our experience with MLCommons, educational initiatives, and programs like the DOE's Trillion Parameter Consortium, key barriers include high resource demands, limited access to specialized hardware, lack of benchmark design expertise, and uncertainty in relating results to application domains. Current benchmarks often emphasize peak performance on top-tier hardware, offering limited guidance for diverse, real-world scenarios.
Benchmarking must become dynamic, incorporating evolving models, updated data, and heterogeneous platforms while maintaining transparency, reproducibility, and interpretability. Democratization requires both technical innovation and systematic education across levels, building sustained expertise in benchmark design and use. Benchmarks should support application-relevant comparisons, enabling informed, context-sensitive decisions. Dynamic, inclusive benchmarking will ensure evaluation keeps pace with AI evolution and supports responsible, reproducible, and accessible AI deployment. Community efforts can provide a foundation for AI Benchmark Carpentry. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_11588 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AI Benchmark Democratization and Carpentry von Laszewski, Gregor Brewer, Wesley Thiyagalingam, Jeyan Papay, Juri Foundjem, Armstrong Luszczek, Piotr Emani, Murali Moore, Shirley V. Reddi, Vijay Janapa Sinclair, Matthew D. Lobentanzer, Sebastian Goswami, Sujata Hawks, Benjamin Colombo, Marco Tran, Nhan Kirkpatrick, Christine R. Alsudais, Abdulkareem Barrett, Gregg Li, Tianhao Morehouse, Kirsten Venkataraman, Shivaram Jain, Rutwik Mathur, Kartik Lu, Victor Singh, Tejinder Mirza, Khojasteh Z. Chen, Kongtao Kunapuli, Sasidhar Farrell, Gavin Umeton, Renato Fox, Geoffrey C. Artificial Intelligence I.2.6 Benchmarks are a cornerstone of modern machine learning, enabling reproducibility, comparison, and scientific progress. However, AI benchmarks are increasingly complex, requiring dynamic, AI-focused workflows. Rapid evolution in model architectures, scale, datasets, and deployment contexts makes evaluation a moving target. Large language models often memorize static benchmarks, causing a gap between benchmark results and real-world performance. Beyond traditional static benchmarks, continuous adaptive benchmarking frameworks are needed to align scientific assessment with deployment risks. This calls for skills and education in AI Benchmark Carpentry. From our experience with MLCommons, educational initiatives, and programs like the DOE's Trillion Parameter Consortium, key barriers include high resource demands, limited access to specialized hardware, lack of benchmark design expertise, and uncertainty in relating results to application domains. Current benchmarks often emphasize peak performance on top-tier hardware, offering limited guidance for diverse, real-world scenarios. Benchmarking must become dynamic, incorporating evolving models, updated data, and heterogeneous platforms while maintaining transparency, reproducibility, and interpretability. Democratization requires both technical innovation and systematic education across levels, building sustained expertise in benchmark design and use. Benchmarks should support application-relevant comparisons, enabling informed, context-sensitive decisions. Dynamic, inclusive benchmarking will ensure evaluation keeps pace with AI evolution and supports responsible, reproducible, and accessible AI deployment. Community efforts can provide a foundation for AI Benchmark Carpentry. |
| title | AI Benchmark Democratization and Carpentry |
| topic | Artificial Intelligence I.2.6 |
| url | https://arxiv.org/abs/2512.11588 |