Evaluating Large Language Models in Scientific Discovery
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915990269329408 |
|---|---|
| author | Song, Zhangde Lu, Jieyu Du, Yuanqi Yu, Botao Pruyn, Thomas M. Huang, Yue Guo, Kehan Luo, Xiuzhe Qu, Yuanhao Qu, Yi Wang, Yinkai Wang, Haorui Guo, Jeff Gan, Jingru Shojaee, Parshin Luo, Di Bran, Andres M Li, Gen Zhao, Qiyuan Luo, Shao-Xiong Lennon Zhang, Yuxuan Zou, Xiang Zhao, Wanru Zhang, Yifan F. Zhang, Wucheng Zheng, Shunan Zhang, Saiyang Khan, Sartaaj Takrim Rajabi-Kochi, Mahyar Paradi-Maropakis, Samantha Baltoiu, Tony Xie, Fengyu Chen, Tianyang Huang, Kexin Luo, Weiliang Fang, Meijing Yang, Xin Cheng, Lixue He, Jiajun Hassoun, Soha Zhang, Xiangliang Wang, Wei Reddy, Chandan K. Zhang, Chao Zheng, Zhiling Wang, Mengdi Cong, Le Gomes, Carla P. Hsieh, Chang-Yu Nandy, Aditya Schwaller, Philippe Kulik, Heather J. Jia, Haojun Sun, Huan Moosavi, Seyed Mohamad Duan, Chenru |
| author_facet | Song, Zhangde Lu, Jieyu Du, Yuanqi Yu, Botao Pruyn, Thomas M. Huang, Yue Guo, Kehan Luo, Xiuzhe Qu, Yuanhao Qu, Yi Wang, Yinkai Wang, Haorui Guo, Jeff Gan, Jingru Shojaee, Parshin Luo, Di Bran, Andres M Li, Gen Zhao, Qiyuan Luo, Shao-Xiong Lennon Zhang, Yuxuan Zou, Xiang Zhao, Wanru Zhang, Yifan F. Zhang, Wucheng Zheng, Shunan Zhang, Saiyang Khan, Sartaaj Takrim Rajabi-Kochi, Mahyar Paradi-Maropakis, Samantha Baltoiu, Tony Xie, Fengyu Chen, Tianyang Huang, Kexin Luo, Weiliang Fang, Meijing Yang, Xin Cheng, Lixue He, Jiajun Hassoun, Soha Zhang, Xiangliang Wang, Wei Reddy, Chandan K. Zhang, Chao Zheng, Zhiling Wang, Mengdi Cong, Le Gomes, Carla P. Hsieh, Chang-Yu Nandy, Aditya Schwaller, Philippe Kulik, Heather J. Jia, Haojun Sun, Huan Moosavi, Seyed Mohamad Duan, Chenru |
| contents | Large language models (LLMs) are increasingly applied to scientific research, yet prevailing science benchmarks probe decontextualized knowledge and overlook the iterative reasoning, hypothesis generation, and observation interpretation that drive scientific discovery. We introduce a scenario-grounded benchmark that evaluates LLMs across biology, chemistry, materials, and physics, where domain experts define research projects of genuine interest and decompose them into modular research scenarios from which vetted questions are sampled. The framework assesses models at two levels: (i) question-level accuracy on scenario-tied items and (ii) project-level performance, where models must propose testable hypotheses, design simulations or experiments, and interpret results. Applying this two-phase scientific discovery evaluation (SDE) framework to state-of-the-art LLMs reveals a consistent performance gap relative to general science benchmarks, diminishing return of scaling up model sizes and reasoning, and systematic weaknesses shared across top-tier models from different providers. Large performance variation in research scenarios leads to changing choices of the best performing model on scientific discovery projects evaluated, suggesting all current LLMs are distant to general scientific "superintelligence". Nevertheless, LLMs already demonstrate promise in a great variety of scientific discovery projects, including cases where constituent scenario scores are low, highlighting the role of guided exploration and serendipity in discovery. This SDE framework offers a reproducible benchmark for discovery-relevant evaluation of LLMs and charts practical paths to advance their development toward scientific discovery. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_15567 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Evaluating Large Language Models in Scientific Discovery Song, Zhangde Lu, Jieyu Du, Yuanqi Yu, Botao Pruyn, Thomas M. Huang, Yue Guo, Kehan Luo, Xiuzhe Qu, Yuanhao Qu, Yi Wang, Yinkai Wang, Haorui Guo, Jeff Gan, Jingru Shojaee, Parshin Luo, Di Bran, Andres M Li, Gen Zhao, Qiyuan Luo, Shao-Xiong Lennon Zhang, Yuxuan Zou, Xiang Zhao, Wanru Zhang, Yifan F. Zhang, Wucheng Zheng, Shunan Zhang, Saiyang Khan, Sartaaj Takrim Rajabi-Kochi, Mahyar Paradi-Maropakis, Samantha Baltoiu, Tony Xie, Fengyu Chen, Tianyang Huang, Kexin Luo, Weiliang Fang, Meijing Yang, Xin Cheng, Lixue He, Jiajun Hassoun, Soha Zhang, Xiangliang Wang, Wei Reddy, Chandan K. Zhang, Chao Zheng, Zhiling Wang, Mengdi Cong, Le Gomes, Carla P. Hsieh, Chang-Yu Nandy, Aditya Schwaller, Philippe Kulik, Heather J. Jia, Haojun Sun, Huan Moosavi, Seyed Mohamad Duan, Chenru Artificial Intelligence Materials Science Machine Learning Chemical Physics Large language models (LLMs) are increasingly applied to scientific research, yet prevailing science benchmarks probe decontextualized knowledge and overlook the iterative reasoning, hypothesis generation, and observation interpretation that drive scientific discovery. We introduce a scenario-grounded benchmark that evaluates LLMs across biology, chemistry, materials, and physics, where domain experts define research projects of genuine interest and decompose them into modular research scenarios from which vetted questions are sampled. The framework assesses models at two levels: (i) question-level accuracy on scenario-tied items and (ii) project-level performance, where models must propose testable hypotheses, design simulations or experiments, and interpret results. Applying this two-phase scientific discovery evaluation (SDE) framework to state-of-the-art LLMs reveals a consistent performance gap relative to general science benchmarks, diminishing return of scaling up model sizes and reasoning, and systematic weaknesses shared across top-tier models from different providers. Large performance variation in research scenarios leads to changing choices of the best performing model on scientific discovery projects evaluated, suggesting all current LLMs are distant to general scientific "superintelligence". Nevertheless, LLMs already demonstrate promise in a great variety of scientific discovery projects, including cases where constituent scenario scores are low, highlighting the role of guided exploration and serendipity in discovery. This SDE framework offers a reproducible benchmark for discovery-relevant evaluation of LLMs and charts practical paths to advance their development toward scientific discovery. |
| title | Evaluating Large Language Models in Scientific Discovery |
| topic | Artificial Intelligence Materials Science Machine Learning Chemical Physics |
| url | https://arxiv.org/abs/2512.15567 |