Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.23982 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915313089511424 |
|---|---|
| author | Cheung, Jerry Junyang Shen, Shiyao Zhuang, Yuchen Li, Yinghao Ramprasad, Rampi Zhang, Chao |
| author_facet | Cheung, Jerry Junyang Shen, Shiyao Zhuang, Yuchen Li, Yinghao Ramprasad, Rampi Zhang, Chao |
| contents | Despite recent advances in large language models (LLMs) for materials science, there is a lack of benchmarks for evaluating their domain-specific knowledge and complex reasoning abilities. To bridge this gap, we introduce MSQA, a comprehensive evaluation benchmark of 1,757 graduate-level materials science questions in two formats: detailed explanatory responses and binary True/False assessments. MSQA distinctively challenges LLMs by requiring both precise factual knowledge and multi-step reasoning across seven materials science sub-fields, such as structure-property relationships, synthesis processes, and computational modeling. Through experiments with 10 state-of-the-art LLMs, we identify significant gaps in current LLM performance. While API-based proprietary LLMs achieve up to 84.5% accuracy, open-source (OSS) LLMs peak around 60.5%, and domain-specific LLMs often underperform significantly due to overfitting and distributional shifts. MSQA represents the first benchmark to jointly evaluate the factual and reasoning capabilities of LLMs crucial for LLMs in advanced materials science. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_23982 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MSQA: Benchmarking LLMs on Graduate-Level Materials Science Reasoning and Knowledge Cheung, Jerry Junyang Shen, Shiyao Zhuang, Yuchen Li, Yinghao Ramprasad, Rampi Zhang, Chao Artificial Intelligence Despite recent advances in large language models (LLMs) for materials science, there is a lack of benchmarks for evaluating their domain-specific knowledge and complex reasoning abilities. To bridge this gap, we introduce MSQA, a comprehensive evaluation benchmark of 1,757 graduate-level materials science questions in two formats: detailed explanatory responses and binary True/False assessments. MSQA distinctively challenges LLMs by requiring both precise factual knowledge and multi-step reasoning across seven materials science sub-fields, such as structure-property relationships, synthesis processes, and computational modeling. Through experiments with 10 state-of-the-art LLMs, we identify significant gaps in current LLM performance. While API-based proprietary LLMs achieve up to 84.5% accuracy, open-source (OSS) LLMs peak around 60.5%, and domain-specific LLMs often underperform significantly due to overfitting and distributional shifts. MSQA represents the first benchmark to jointly evaluate the factual and reasoning capabilities of LLMs crucial for LLMs in advanced materials science. |
| title | MSQA: Benchmarking LLMs on Graduate-Level Materials Science Reasoning and Knowledge |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2505.23982 |