SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914078152196096 |
|---|---|
| author | Feng, Kehua Shen, Xinyi Wang, Weijie Zhuang, Xiang Tang, Yuqi Zhang, Qiang Ding, Keyan |
| author_facet | Feng, Kehua Shen, Xinyi Wang, Weijie Zhuang, Xiang Tang, Yuqi Zhang, Qiang Ding, Keyan |
| contents | Large language models (LLMs) are playing an increasingly important role in scientific research, yet there remains a lack of comprehensive benchmarks to evaluate the breadth and depth of scientific knowledge embedded in these models. To address this gap, we introduce SciKnowEval, a large-scale dataset designed to systematically assess LLMs across five progressive levels of scientific understanding: memory, comprehension, reasoning, discernment, and application. SciKnowEval comprises 28K multi-level questions and solutions spanning biology, chemistry, physics, and materials science. Using this benchmark, we evaluate 20 leading open-source and proprietary LLMs. The results show that while proprietary models often achieve state-of-the-art performance, substantial challenges remain -- particularly in scientific reasoning and real-world application. We envision SciKnowEval as a standard benchmark for evaluating scientific capabilities in LLMs and as a catalyst for advancing more capable and reliable scientific language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_09098 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models Feng, Kehua Shen, Xinyi Wang, Weijie Zhuang, Xiang Tang, Yuqi Zhang, Qiang Ding, Keyan Computation and Language Large language models (LLMs) are playing an increasingly important role in scientific research, yet there remains a lack of comprehensive benchmarks to evaluate the breadth and depth of scientific knowledge embedded in these models. To address this gap, we introduce SciKnowEval, a large-scale dataset designed to systematically assess LLMs across five progressive levels of scientific understanding: memory, comprehension, reasoning, discernment, and application. SciKnowEval comprises 28K multi-level questions and solutions spanning biology, chemistry, physics, and materials science. Using this benchmark, we evaluate 20 leading open-source and proprietary LLMs. The results show that while proprietary models often achieve state-of-the-art performance, substantial challenges remain -- particularly in scientific reasoning and real-world application. We envision SciKnowEval as a standard benchmark for evaluating scientific capabilities in LLMs and as a catalyst for advancing more capable and reliable scientific language models. |
| title | SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.09098 |