Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2512.24867 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915709520445440 |
|---|---|
| author | Liang, Yiming Li, Yizhi Du, Yantao Zhang, Ge Zhou, Jiayi Wu, Yuchen Piao, Yinzhu Cao, Denghui Sun, Tong Li, Ziniu Du, Li Lei, Bo Liu, Jiaheng Lin, Chenghua Zhang, Zhaoxiang Huang, Wenhao Zhang, Jiajun |
| author_facet | Liang, Yiming Li, Yizhi Du, Yantao Zhang, Ge Zhou, Jiayi Wu, Yuchen Piao, Yinzhu Cao, Denghui Sun, Tong Li, Ziniu Du, Li Lei, Bo Liu, Jiaheng Lin, Chenghua Zhang, Zhaoxiang Huang, Wenhao Zhang, Jiajun |
| contents | Benchmarks play a crucial role in tracking the rapid advancement of large language models (LLMs) and identifying their capability boundaries. However, existing benchmarks predominantly curate questions at the question level, suffering from three fundamental limitations: vulnerability to data contamination, restriction to single-knowledge-point assessment, and reliance on costly domain expert annotation. We propose Encyclo-K, a statement-based benchmark that rethinks benchmark construction from the ground up. Our key insight is that knowledge statements, not questions, can serve as the unit of curation, and questions can then be constructed from them. We extract standalone knowledge statements from authoritative textbooks and dynamically compose them into evaluation questions through random sampling at test time. This design directly addresses all three limitations: the combinatorial space is too vast to memorize, and model rankings remain stable across dynamically generated question sets, enabling reliable periodic dataset refresh; each question aggregates 8-10 statements for comprehensive multi-knowledge assessment; annotators only verify formatting compliance without requiring domain expertise, substantially reducing annotation costs. Experiments on over 50 LLMs demonstrate that Encyclo-K poses substantial challenges with strong discriminative power. Even the top-performing OpenAI-GPT-5.1 achieves only 62.07% accuracy, and model performance displays a clear gradient distribution--reasoning models span from 16.04% to 62.07%, while chat models range from 9.71% to 50.40%. These results validate the challenges introduced by dynamic evaluation and multi-statement comprehensive understanding. These findings establish Encyclo-K as a scalable framework for dynamic evaluation of LLMs' comprehensive understanding over multiple fine-grained disciplinary knowledge statements. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_24867 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements Liang, Yiming Li, Yizhi Du, Yantao Zhang, Ge Zhou, Jiayi Wu, Yuchen Piao, Yinzhu Cao, Denghui Sun, Tong Li, Ziniu Du, Li Lei, Bo Liu, Jiaheng Lin, Chenghua Zhang, Zhaoxiang Huang, Wenhao Zhang, Jiajun Computation and Language Artificial Intelligence Benchmarks play a crucial role in tracking the rapid advancement of large language models (LLMs) and identifying their capability boundaries. However, existing benchmarks predominantly curate questions at the question level, suffering from three fundamental limitations: vulnerability to data contamination, restriction to single-knowledge-point assessment, and reliance on costly domain expert annotation. We propose Encyclo-K, a statement-based benchmark that rethinks benchmark construction from the ground up. Our key insight is that knowledge statements, not questions, can serve as the unit of curation, and questions can then be constructed from them. We extract standalone knowledge statements from authoritative textbooks and dynamically compose them into evaluation questions through random sampling at test time. This design directly addresses all three limitations: the combinatorial space is too vast to memorize, and model rankings remain stable across dynamically generated question sets, enabling reliable periodic dataset refresh; each question aggregates 8-10 statements for comprehensive multi-knowledge assessment; annotators only verify formatting compliance without requiring domain expertise, substantially reducing annotation costs. Experiments on over 50 LLMs demonstrate that Encyclo-K poses substantial challenges with strong discriminative power. Even the top-performing OpenAI-GPT-5.1 achieves only 62.07% accuracy, and model performance displays a clear gradient distribution--reasoning models span from 16.04% to 62.07%, while chat models range from 9.71% to 50.40%. These results validate the challenges introduced by dynamic evaluation and multi-statement comprehensive understanding. These findings establish Encyclo-K as a scalable framework for dynamic evaluation of LLMs' comprehensive understanding over multiple fine-grained disciplinary knowledge statements. |
| title | Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2512.24867 |