Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2306.05783 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909132961873920 |
|---|---|
| author | Gu, Zhouhong Zhu, Xiaoxuan Ye, Haoning Zhang, Lin Wang, Jianchen Zhu, Yixin Jiang, Sihang Xiong, Zhuozhi Li, Zihan Wu, Weijie He, Qianyu Xu, Rui Huang, Wenhao Liu, Jingping Wang, Zili Wang, Shusen Zheng, Weiguo Feng, Hongwei Xiao, Yanghua |
| author_facet | Gu, Zhouhong Zhu, Xiaoxuan Ye, Haoning Zhang, Lin Wang, Jianchen Zhu, Yixin Jiang, Sihang Xiong, Zhuozhi Li, Zihan Wu, Weijie He, Qianyu Xu, Rui Huang, Wenhao Liu, Jingping Wang, Zili Wang, Shusen Zheng, Weiguo Feng, Hongwei Xiao, Yanghua |
| contents | New Natural Langauge Process~(NLP) benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present Xiezhi, the most comprehensive evaluation suite designed to assess holistic domain knowledge. Xiezhi comprises multiple-choice questions across 516 diverse disciplines ranging from 13 different subjects with 249,587 questions and accompanied by Xiezhi-Specialty and Xiezhi-Interdiscipline, both with 15k questions. We conduct evaluation of the 47 cutting-edge LLMs on Xiezhi. Results indicate that LLMs exceed average performance of humans in science, engineering, agronomy, medicine, and art, but fall short in economics, jurisprudence, pedagogy, literature, history, and management. We anticipate Xiezhi will help analyze important strengths and shortcomings of LLMs, and the benchmark is released in~\url{https://github.com/MikeGu721/XiezhiBenchmark}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2306_05783 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation Gu, Zhouhong Zhu, Xiaoxuan Ye, Haoning Zhang, Lin Wang, Jianchen Zhu, Yixin Jiang, Sihang Xiong, Zhuozhi Li, Zihan Wu, Weijie He, Qianyu Xu, Rui Huang, Wenhao Liu, Jingping Wang, Zili Wang, Shusen Zheng, Weiguo Feng, Hongwei Xiao, Yanghua Computation and Language New Natural Langauge Process~(NLP) benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present Xiezhi, the most comprehensive evaluation suite designed to assess holistic domain knowledge. Xiezhi comprises multiple-choice questions across 516 diverse disciplines ranging from 13 different subjects with 249,587 questions and accompanied by Xiezhi-Specialty and Xiezhi-Interdiscipline, both with 15k questions. We conduct evaluation of the 47 cutting-edge LLMs on Xiezhi. Results indicate that LLMs exceed average performance of humans in science, engineering, agronomy, medicine, and art, but fall short in economics, jurisprudence, pedagogy, literature, history, and management. We anticipate Xiezhi will help analyze important strengths and shortcomings of LLMs, and the benchmark is released in~\url{https://github.com/MikeGu721/XiezhiBenchmark}. |
| title | Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2306.05783 |