Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2509.26574 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917473562918912 |
|---|---|
| author | Zhu, Minhui Tian, Minyang Yang, Xiaocheng Zhou, Tianci Yuan, Lifan Zhu, Penghao Chertkov, Eli Liu, Shengyan Du, Yufeng Ji, Ziming Das, Indranil Chen, Qingzhi Cao, Junyi Du, Yufeng Yu, Jiabin Wu, Peixue He, Jinchen Su, Yifan Jiang, Yikun Zhang, Yujie Liu, Chang Huang, Ze-Min Jia, Weizhen Wang, Yunkai Jafarpour, Farshid Zhao, Yong Chen, Xinan Shelton, Jessie Young, Aaron W. Bartolotta, John Xu, Wenchao Sun, Yue Chu, Anjun Colussi, Victor Akers, Chris Brooks, Nathan Fu, Wenbo Zhao, Jinchao Qi, Marvin Mu, Anqi Yang, Yubo Zang, Allen Lyu, Yang Mai, Peizhi Wilson, Christopher Guo, Xuefei Zhou, Juntai Inafuku, Daniel Xue, Chi Gao, Luyu Yang, Ze Hein, Yaïr Kahn, Yonatan Zhou, Kevin Luo, Di Wilson, John Drew Reilly, Jarrod T. Bandak, Dmytro Press, Ofir Yang, Liang Wang, Xueying Tong, Hao Chia, Nicolas Huerta, Eliu Peng, Hao |
| author_facet | Zhu, Minhui Tian, Minyang Yang, Xiaocheng Zhou, Tianci Yuan, Lifan Zhu, Penghao Chertkov, Eli Liu, Shengyan Du, Yufeng Ji, Ziming Das, Indranil Chen, Qingzhi Cao, Junyi Du, Yufeng Yu, Jiabin Wu, Peixue He, Jinchen Su, Yifan Jiang, Yikun Zhang, Yujie Liu, Chang Huang, Ze-Min Jia, Weizhen Wang, Yunkai Jafarpour, Farshid Zhao, Yong Chen, Xinan Shelton, Jessie Young, Aaron W. Bartolotta, John Xu, Wenchao Sun, Yue Chu, Anjun Colussi, Victor Akers, Chris Brooks, Nathan Fu, Wenbo Zhao, Jinchao Qi, Marvin Mu, Anqi Yang, Yubo Zang, Allen Lyu, Yang Mai, Peizhi Wilson, Christopher Guo, Xuefei Zhou, Juntai Inafuku, Daniel Xue, Chi Gao, Luyu Yang, Ze Hein, Yaïr Kahn, Yonatan Zhou, Kevin Luo, Di Wilson, John Drew Reilly, Jarrod T. Bandak, Dmytro Press, Ofir Yang, Liang Wang, Xueying Tong, Hao Chia, Nicolas Huerta, Eliu Peng, Hao |
| contents | While large language models (LLMs) with reasoning capabilities are progressing rapidly on high-school math competitions and coding, can they reason effectively through complex, open-ended challenges found in frontier physics research? And crucially, what kinds of reasoning tasks do physicists want LLMs to assist with? To address these questions, we present the CritPt (Complex Research using Integrated Thinking - Physics Test, pronounced "critical point"), the first benchmark designed to test LLMs on unpublished, research-level reasoning tasks that broadly covers modern physics research areas, including condensed matter, quantum physics, atomic, molecular & optical physics, astrophysics, high energy physics, mathematical physics, statistical physics, nuclear physics, nonlinear dynamics, fluid dynamics and biophysics. CritPt consists of 71 composite research challenges designed to simulate full-scale research projects at the entry level, which are also decomposed to 190 simpler checkpoint tasks for more fine-grained insights. All problems are newly created by 50+ active physics researchers based on their own research. Every problem is hand-curated to admit a guess-resistant and machine-verifiable answer and is evaluated by an automated grading pipeline heavily customized for advanced physics-specific output formats. We find that while current state-of-the-art LLMs show early promise on isolated checkpoints, they remain far from being able to reliably solve full research-scale challenges: the best average accuracy among base models is only 5.7%, achieved by GPT-5 (high), moderately rising to around 10% when equipped with coding tools. Through the realistic yet standardized evaluation offered by CritPt, we highlight a large disconnect between current model capabilities and realistic physics research demands, offering a foundation to guide the development of scientifically grounded AI tools. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_26574 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark Zhu, Minhui Tian, Minyang Yang, Xiaocheng Zhou, Tianci Yuan, Lifan Zhu, Penghao Chertkov, Eli Liu, Shengyan Du, Yufeng Ji, Ziming Das, Indranil Chen, Qingzhi Cao, Junyi Du, Yufeng Yu, Jiabin Wu, Peixue He, Jinchen Su, Yifan Jiang, Yikun Zhang, Yujie Liu, Chang Huang, Ze-Min Jia, Weizhen Wang, Yunkai Jafarpour, Farshid Zhao, Yong Chen, Xinan Shelton, Jessie Young, Aaron W. Bartolotta, John Xu, Wenchao Sun, Yue Chu, Anjun Colussi, Victor Akers, Chris Brooks, Nathan Fu, Wenbo Zhao, Jinchao Qi, Marvin Mu, Anqi Yang, Yubo Zang, Allen Lyu, Yang Mai, Peizhi Wilson, Christopher Guo, Xuefei Zhou, Juntai Inafuku, Daniel Xue, Chi Gao, Luyu Yang, Ze Hein, Yaïr Kahn, Yonatan Zhou, Kevin Luo, Di Wilson, John Drew Reilly, Jarrod T. Bandak, Dmytro Press, Ofir Yang, Liang Wang, Xueying Tong, Hao Chia, Nicolas Huerta, Eliu Peng, Hao Artificial Intelligence Other Condensed Matter Computation and Language High Energy Physics - Theory Quantum Physics While large language models (LLMs) with reasoning capabilities are progressing rapidly on high-school math competitions and coding, can they reason effectively through complex, open-ended challenges found in frontier physics research? And crucially, what kinds of reasoning tasks do physicists want LLMs to assist with? To address these questions, we present the CritPt (Complex Research using Integrated Thinking - Physics Test, pronounced "critical point"), the first benchmark designed to test LLMs on unpublished, research-level reasoning tasks that broadly covers modern physics research areas, including condensed matter, quantum physics, atomic, molecular & optical physics, astrophysics, high energy physics, mathematical physics, statistical physics, nuclear physics, nonlinear dynamics, fluid dynamics and biophysics. CritPt consists of 71 composite research challenges designed to simulate full-scale research projects at the entry level, which are also decomposed to 190 simpler checkpoint tasks for more fine-grained insights. All problems are newly created by 50+ active physics researchers based on their own research. Every problem is hand-curated to admit a guess-resistant and machine-verifiable answer and is evaluated by an automated grading pipeline heavily customized for advanced physics-specific output formats. We find that while current state-of-the-art LLMs show early promise on isolated checkpoints, they remain far from being able to reliably solve full research-scale challenges: the best average accuracy among base models is only 5.7%, achieved by GPT-5 (high), moderately rising to around 10% when equipped with coding tools. Through the realistic yet standardized evaluation offered by CritPt, we highlight a large disconnect between current model capabilities and realistic physics research demands, offering a foundation to guide the development of scientifically grounded AI tools. |
| title | Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark |
| topic | Artificial Intelligence Other Condensed Matter Computation and Language High Energy Physics - Theory Quantum Physics |
| url | https://arxiv.org/abs/2509.26574 |