PRBench: End-to-end Paper Reproduction in Physics Research
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912986866647040 |
|---|---|
| author | Qiu, Shi Deng, Junyi Deng, Yiwei Dong, Haoran Fu, Jieyu Li, Mao Li, Zeyu Zhang, Zhaolong Zheng, Huiwen Bao, Leidong Lv, Anqi Mo, Zihan Niu, Yadi Peng, Yiyang Tian, Yu Wang, Yili Wang, Ziyu Wang, Zi-Yu Wei, Jiashen Wu, Liuheng Xue, Aoran Yang, Leyi Yuan, Guanglu Zhan, Xiarui Zhang, Jingjun Zheng, Zifan Liu, Pengfei Zhen, Linrui Li, Kaiyang Li, Qichang Zhou, Ziheng Nian, Guo-En Xiao, Yunwei Cao, Qing-Hong Dai, Linjie Feng, Xu Gao, Peng Gu, Ying Liu, Chang Liu, Jia Luo, Ming-xing Ma, Yan-Qing Peng, Liang-You Song, Huichao Wang, Shufeng Wang, Chenxu Wang, Tao Wang, Yi-Nan Wu, Chengyin Zhao, Pengwei Zhu, Hua Xing |
| author_facet | Qiu, Shi Deng, Junyi Deng, Yiwei Dong, Haoran Fu, Jieyu Li, Mao Li, Zeyu Zhang, Zhaolong Zheng, Huiwen Bao, Leidong Lv, Anqi Mo, Zihan Niu, Yadi Peng, Yiyang Tian, Yu Wang, Yili Wang, Ziyu Wang, Zi-Yu Wei, Jiashen Wu, Liuheng Xue, Aoran Yang, Leyi Yuan, Guanglu Zhan, Xiarui Zhang, Jingjun Zheng, Zifan Liu, Pengfei Zhen, Linrui Li, Kaiyang Li, Qichang Zhou, Ziheng Nian, Guo-En Xiao, Yunwei Cao, Qing-Hong Dai, Linjie Feng, Xu Gao, Peng Gu, Ying Liu, Chang Liu, Jia Luo, Ming-xing Ma, Yan-Qing Peng, Liang-You Song, Huichao Wang, Shufeng Wang, Chenxu Wang, Tao Wang, Yi-Nan Wu, Chengyin Zhao, Pengwei Zhu, Hua Xing |
| contents | AI agents powered by large language models exhibit strong reasoning and problem-solving capabilities, enabling them to assist scientific research tasks such as formula derivation and code generation. However, whether these agents can reliably perform end-to-end reproduction from real scientific papers remains an open question. We introduce PRBench, a benchmark of 30 expert-curated tasks spanning 11 subfields of physics. Each task requires an agent to comprehend the methodology of a published paper, implement the corresponding algorithms from scratch, and produce quantitative results matching the original publication. Agents are provided only with the task instruction and paper content, and operate in a sandboxed execution environment. All tasks are contributed by domain experts from over 20 research groups at the School of Physics, Peking University, each grounded in a real published paper and validated through end-to-end reproduction with verified ground-truth results and detailed scoring rubrics. Using an agentified assessment pipeline, we evaluate a set of coding agents on PRBench and analyze their capabilities across key dimensions of scientific reasoning and execution. The best-performing agent, OpenAI Codex powered by GPT-5.3-Codex, achieves a mean overall score of 34%. All agents exhibit a zero end-to-end callback success rate, with particularly poor performance in data accuracy and code correctness. We further identify systematic failure modes, including errors in formula implementation, inability to debug numerical simulations, and fabrication of output data. Overall, PRBench provides a rigorous benchmark for evaluating progress toward autonomous scientific research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_27646 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | PRBench: End-to-end Paper Reproduction in Physics Research Qiu, Shi Deng, Junyi Deng, Yiwei Dong, Haoran Fu, Jieyu Li, Mao Li, Zeyu Zhang, Zhaolong Zheng, Huiwen Bao, Leidong Lv, Anqi Mo, Zihan Niu, Yadi Peng, Yiyang Tian, Yu Wang, Yili Wang, Ziyu Wang, Zi-Yu Wei, Jiashen Wu, Liuheng Xue, Aoran Yang, Leyi Yuan, Guanglu Zhan, Xiarui Zhang, Jingjun Zheng, Zifan Liu, Pengfei Zhen, Linrui Li, Kaiyang Li, Qichang Zhou, Ziheng Nian, Guo-En Xiao, Yunwei Cao, Qing-Hong Dai, Linjie Feng, Xu Gao, Peng Gu, Ying Liu, Chang Liu, Jia Luo, Ming-xing Ma, Yan-Qing Peng, Liang-You Song, Huichao Wang, Shufeng Wang, Chenxu Wang, Tao Wang, Yi-Nan Wu, Chengyin Zhao, Pengwei Zhu, Hua Xing Computation and Language High Energy Physics - Lattice High Energy Physics - Phenomenology Computational Physics Optics AI agents powered by large language models exhibit strong reasoning and problem-solving capabilities, enabling them to assist scientific research tasks such as formula derivation and code generation. However, whether these agents can reliably perform end-to-end reproduction from real scientific papers remains an open question. We introduce PRBench, a benchmark of 30 expert-curated tasks spanning 11 subfields of physics. Each task requires an agent to comprehend the methodology of a published paper, implement the corresponding algorithms from scratch, and produce quantitative results matching the original publication. Agents are provided only with the task instruction and paper content, and operate in a sandboxed execution environment. All tasks are contributed by domain experts from over 20 research groups at the School of Physics, Peking University, each grounded in a real published paper and validated through end-to-end reproduction with verified ground-truth results and detailed scoring rubrics. Using an agentified assessment pipeline, we evaluate a set of coding agents on PRBench and analyze their capabilities across key dimensions of scientific reasoning and execution. The best-performing agent, OpenAI Codex powered by GPT-5.3-Codex, achieves a mean overall score of 34%. All agents exhibit a zero end-to-end callback success rate, with particularly poor performance in data accuracy and code correctness. We further identify systematic failure modes, including errors in formula implementation, inability to debug numerical simulations, and fabrication of output data. Overall, PRBench provides a rigorous benchmark for evaluating progress toward autonomous scientific research. |
| title | PRBench: End-to-end Paper Reproduction in Physics Research |
| topic | Computation and Language High Energy Physics - Lattice High Energy Physics - Phenomenology Computational Physics Optics |
| url | https://arxiv.org/abs/2603.27646 |