CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916634519666688 |
|---|---|
| author | Song, Xiaoshuai Diao, Muxi Dong, Guanting Wang, Zhengyang Fu, Yujia Qiao, Runqi Wang, Zhexu Fu, Dayuan Wu, Huangxuan Liang, Bin Zeng, Weihao Wang, Yejie GongQue, Zhuoma Yu, Jianing Tan, Qiuna Xu, Weiran |
| author_facet | Song, Xiaoshuai Diao, Muxi Dong, Guanting Wang, Zhengyang Fu, Yujia Qiao, Runqi Wang, Zhexu Fu, Dayuan Wu, Huangxuan Liang, Bin Zeng, Weihao Wang, Yejie GongQue, Zhuoma Yu, Jianing Tan, Qiuna Xu, Weiran |
| contents | Large language models (LLMs) have demonstrated significant potential in advancing various fields of research and society. However, the current community of LLMs overly focuses on benchmarks for analyzing specific foundational skills (e.g. mathematics and code generation), neglecting an all-round evaluation of the computer science field. To bridge this gap, we introduce CS-Bench, the first multilingual (English, Chinese, French, German) benchmark dedicated to evaluating the performance of LLMs in computer science. CS-Bench comprises approximately 10K meticulously curated test samples, covering 26 subfields across 4 key areas of computer science, encompassing various task forms and divisions of knowledge and reasoning. Utilizing CS-Bench, we conduct a comprehensive evaluation of over 30 mainstream LLMs, revealing the relationship between CS performance and model scales. We also quantitatively analyze the reasons for failures in existing LLMs and highlight directions for improvements, including knowledge supplementation and CS-specific reasoning. Further cross-capability experiments show a high correlation between LLMs' capabilities in computer science and their abilities in mathematics and coding. Moreover, expert LLMs specialized in mathematics and coding also demonstrate strong performances in several CS subfields. Looking ahead, we envision CS-Bench serving as a cornerstone for LLM applications in the CS field and paving new avenues in assessing LLMs' diverse reasoning capabilities. The CS-Bench data and evaluation code are available at https://github.com/csbench/csbench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_08587 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery Song, Xiaoshuai Diao, Muxi Dong, Guanting Wang, Zhengyang Fu, Yujia Qiao, Runqi Wang, Zhexu Fu, Dayuan Wu, Huangxuan Liang, Bin Zeng, Weihao Wang, Yejie GongQue, Zhuoma Yu, Jianing Tan, Qiuna Xu, Weiran Computation and Language Artificial Intelligence Machine Learning Large language models (LLMs) have demonstrated significant potential in advancing various fields of research and society. However, the current community of LLMs overly focuses on benchmarks for analyzing specific foundational skills (e.g. mathematics and code generation), neglecting an all-round evaluation of the computer science field. To bridge this gap, we introduce CS-Bench, the first multilingual (English, Chinese, French, German) benchmark dedicated to evaluating the performance of LLMs in computer science. CS-Bench comprises approximately 10K meticulously curated test samples, covering 26 subfields across 4 key areas of computer science, encompassing various task forms and divisions of knowledge and reasoning. Utilizing CS-Bench, we conduct a comprehensive evaluation of over 30 mainstream LLMs, revealing the relationship between CS performance and model scales. We also quantitatively analyze the reasons for failures in existing LLMs and highlight directions for improvements, including knowledge supplementation and CS-specific reasoning. Further cross-capability experiments show a high correlation between LLMs' capabilities in computer science and their abilities in mathematics and coding. Moreover, expert LLMs specialized in mathematics and coding also demonstrate strong performances in several CS subfields. Looking ahead, we envision CS-Bench serving as a cornerstone for LLM applications in the CS field and paving new avenues in assessing LLMs' diverse reasoning capabilities. The CS-Bench data and evaluation code are available at https://github.com/csbench/csbench. |
| title | CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2406.08587 |