Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2506.12577 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913893922635776 |
|---|---|
| author | Chen, Yongrui Liu, Zhiqiang Yu, Jing Ren, Lin Hu, Nan Dai, Xinbang Liu, Jiajun Kang, Jiazhen Zhang, Shenyu Wang, Xinda Ding, Keyan Shen, Pengfei Zhu, Haolei Deng, Hongjie Wang, Yisong Wu, Tongtong Bi, Sheng Zhang, Wen Wu, Tianxing Ji, Qiu Wang, Haofen Chen, Wenliang Chen, Huajun Qi, Guilin |
| author_facet | Chen, Yongrui Liu, Zhiqiang Yu, Jing Ren, Lin Hu, Nan Dai, Xinbang Liu, Jiajun Kang, Jiazhen Zhang, Shenyu Wang, Xinda Ding, Keyan Shen, Pengfei Zhu, Haolei Deng, Hongjie Wang, Yisong Wu, Tongtong Bi, Sheng Zhang, Wen Wu, Tianxing Ji, Qiu Wang, Haofen Chen, Wenliang Chen, Huajun Qi, Guilin |
| contents | Large Language Models (LLMs) have demonstrated substantial progress on reasoning tasks involving unstructured text, yet their capabilities significantly deteriorate when reasoning requires integrating structured external knowledge such as knowledge graphs, code snippets, or formal logic. This limitation is partly due to the absence of benchmarks capable of systematically evaluating LLM performance across diverse structured knowledge modalities. To address this gap, we introduce \textbf{\textsc{OneEval}}, a comprehensive benchmark explicitly designed to assess the knowledge-intensive reasoning capabilities of LLMs across four structured knowledge modalities, unstructured text, knowledge graphs, code, and formal logic, and five critical domains (general knowledge, government, science, law, and programming). \textsc{OneEval} comprises 4,019 carefully curated instances and includes a challenging subset, \textsc{OneEval}\textsubscript{Hard}, consisting of 1,285 particularly difficult cases. Through extensive evaluation of 18 state-of-the-art open-source and proprietary LLMs, we establish three core findings: a) \emph{persistent limitations in structured reasoning}, with even the strongest model achieving only 32.2\% accuracy on \textsc{OneEval}\textsubscript{Hard}; b) \emph{performance consistently declines as the structural complexity of the knowledge base increases}, with accuracy dropping sharply from 53\% (textual reasoning) to 25\% (formal logic); and c) \emph{diminishing returns from extended reasoning chains}, highlighting the critical need for models to adapt reasoning depth appropriately to task complexity. We release the \textsc{OneEval} datasets, evaluation scripts, and baseline results publicly, accompanied by a leaderboard to facilitate ongoing advancements in structured knowledge reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_12577 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | OneEval: Benchmarking LLM Knowledge-intensive Reasoning over Diverse Knowledge Bases Chen, Yongrui Liu, Zhiqiang Yu, Jing Ren, Lin Hu, Nan Dai, Xinbang Liu, Jiajun Kang, Jiazhen Zhang, Shenyu Wang, Xinda Ding, Keyan Shen, Pengfei Zhu, Haolei Deng, Hongjie Wang, Yisong Wu, Tongtong Bi, Sheng Zhang, Wen Wu, Tianxing Ji, Qiu Wang, Haofen Chen, Wenliang Chen, Huajun Qi, Guilin Computation and Language Large Language Models (LLMs) have demonstrated substantial progress on reasoning tasks involving unstructured text, yet their capabilities significantly deteriorate when reasoning requires integrating structured external knowledge such as knowledge graphs, code snippets, or formal logic. This limitation is partly due to the absence of benchmarks capable of systematically evaluating LLM performance across diverse structured knowledge modalities. To address this gap, we introduce \textbf{\textsc{OneEval}}, a comprehensive benchmark explicitly designed to assess the knowledge-intensive reasoning capabilities of LLMs across four structured knowledge modalities, unstructured text, knowledge graphs, code, and formal logic, and five critical domains (general knowledge, government, science, law, and programming). \textsc{OneEval} comprises 4,019 carefully curated instances and includes a challenging subset, \textsc{OneEval}\textsubscript{Hard}, consisting of 1,285 particularly difficult cases. Through extensive evaluation of 18 state-of-the-art open-source and proprietary LLMs, we establish three core findings: a) \emph{persistent limitations in structured reasoning}, with even the strongest model achieving only 32.2\% accuracy on \textsc{OneEval}\textsubscript{Hard}; b) \emph{performance consistently declines as the structural complexity of the knowledge base increases}, with accuracy dropping sharply from 53\% (textual reasoning) to 25\% (formal logic); and c) \emph{diminishing returns from extended reasoning chains}, highlighting the critical need for models to adapt reasoning depth appropriately to task complexity. We release the \textsc{OneEval} datasets, evaluation scripts, and baseline results publicly, accompanied by a leaderboard to facilitate ongoing advancements in structured knowledge reasoning. |
| title | OneEval: Benchmarking LLM Knowledge-intensive Reasoning over Diverse Knowledge Bases |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.12577 |