Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.21695 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909710638120960 |
|---|---|
| author | Barman, Kristian G. Caron, Sascha Hasibi, Faegheh Shalugin, Eugene Marcet, Yoris Otte, Johannes de Regt, Henk W. Moody, Merijn |
| author_facet | Barman, Kristian G. Caron, Sascha Hasibi, Faegheh Shalugin, Eugene Marcet, Yoris Otte, Johannes de Regt, Henk W. Moody, Merijn |
| contents | We introduce a benchmark framework developed by and for the scientific community to evaluate, monitor and steer large language model development in fundamental physics. Building on philosophical concepts of scientific understanding and creativity, we develop a scoring system in which each question is scored by an expert for its correctness, difficulty, and surprise. The questions are of three forms: (i) multiple-choice questions for conceptual understanding, (ii) analytical problems requiring mathematical derivation, and (iii) openended tasks requiring complex problem solving. Our current dataset contains diverse set of examples, including a machine learning challenge to classify high-energy physics events, such as the four top quark signal. To ensure continued relevance, we propose a living benchmark, where physicists contribute questions, for instance alongside new publications. We invite contributions via: http://www.physicsbenchmarks.org/. We hope that this benchmark will enable a targeted AI development that can make a meaningful contribution to fundamental physics research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_21695 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards a Large Physics Benchmark Barman, Kristian G. Caron, Sascha Hasibi, Faegheh Shalugin, Eugene Marcet, Yoris Otte, Johannes de Regt, Henk W. Moody, Merijn Data Analysis, Statistics and Probability Artificial Intelligence High Energy Physics - Phenomenology Computational Physics History and Philosophy of Physics We introduce a benchmark framework developed by and for the scientific community to evaluate, monitor and steer large language model development in fundamental physics. Building on philosophical concepts of scientific understanding and creativity, we develop a scoring system in which each question is scored by an expert for its correctness, difficulty, and surprise. The questions are of three forms: (i) multiple-choice questions for conceptual understanding, (ii) analytical problems requiring mathematical derivation, and (iii) openended tasks requiring complex problem solving. Our current dataset contains diverse set of examples, including a machine learning challenge to classify high-energy physics events, such as the four top quark signal. To ensure continued relevance, we propose a living benchmark, where physicists contribute questions, for instance alongside new publications. We invite contributions via: http://www.physicsbenchmarks.org/. We hope that this benchmark will enable a targeted AI development that can make a meaningful contribution to fundamental physics research. |
| title | Towards a Large Physics Benchmark |
| topic | Data Analysis, Statistics and Probability Artificial Intelligence High Energy Physics - Phenomenology Computational Physics History and Philosophy of Physics |
| url | https://arxiv.org/abs/2507.21695 |