SciCUEval: A Comprehensive Dataset for Evaluating Scientific Context Understanding in Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918028120162304 |
|---|---|
| author | Yu, Jing Tang, Yuqi Feng, Kehua Rao, Mingyang Liang, Lei Zhang, Zhiqiang Sun, Mengshu Zhang, Wen Zhang, Qiang Ding, Keyan Chen, Huajun |
| author_facet | Yu, Jing Tang, Yuqi Feng, Kehua Rao, Mingyang Liang, Lei Zhang, Zhiqiang Sun, Mengshu Zhang, Wen Zhang, Qiang Ding, Keyan Chen, Huajun |
| contents | Large Language Models (LLMs) have shown impressive capabilities in contextual understanding and reasoning. However, evaluating their performance across diverse scientific domains remains underexplored, as existing benchmarks primarily focus on general domains and fail to capture the intricate complexity of scientific data. To bridge this gap, we construct SciCUEval, a comprehensive benchmark dataset tailored to assess the scientific context understanding capability of LLMs. It comprises ten domain-specific sub-datasets spanning biology, chemistry, physics, biomedicine, and materials science, integrating diverse data modalities including structured tables, knowledge graphs, and unstructured texts. SciCUEval systematically evaluates four core competencies: Relevant information identification, Information-absence detection, Multi-source information integration, and Context-aware inference, through a variety of question formats. We conduct extensive evaluations of state-of-the-art LLMs on SciCUEval, providing a fine-grained analysis of their strengths and limitations in scientific context understanding, and offering valuable insights for the future development of scientific-domain LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_15094 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SciCUEval: A Comprehensive Dataset for Evaluating Scientific Context Understanding in Large Language Models Yu, Jing Tang, Yuqi Feng, Kehua Rao, Mingyang Liang, Lei Zhang, Zhiqiang Sun, Mengshu Zhang, Wen Zhang, Qiang Ding, Keyan Chen, Huajun Computation and Language Large Language Models (LLMs) have shown impressive capabilities in contextual understanding and reasoning. However, evaluating their performance across diverse scientific domains remains underexplored, as existing benchmarks primarily focus on general domains and fail to capture the intricate complexity of scientific data. To bridge this gap, we construct SciCUEval, a comprehensive benchmark dataset tailored to assess the scientific context understanding capability of LLMs. It comprises ten domain-specific sub-datasets spanning biology, chemistry, physics, biomedicine, and materials science, integrating diverse data modalities including structured tables, knowledge graphs, and unstructured texts. SciCUEval systematically evaluates four core competencies: Relevant information identification, Information-absence detection, Multi-source information integration, and Context-aware inference, through a variety of question formats. We conduct extensive evaluations of state-of-the-art LLMs on SciCUEval, providing a fine-grained analysis of their strengths and limitations in scientific context understanding, and offering valuable insights for the future development of scientific-domain LLMs. |
| title | SciCUEval: A Comprehensive Dataset for Evaluating Scientific Context Understanding in Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.15094 |