Are LLMs Ready for Computer Science Education? A Cross-Domain, Cross-Lingual and Cognitive-Level Evaluation Using Professional Certification Exams
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917391647113216 |
|---|---|
| author | Gao, Chen Liu, Chi Luo, Zhengquan Xiao, Dongfu Sui, Maiying Shen, Sheng Zhu, Congcong Chen, Huajie Zuo, Xuhan Ge, Zongyuan Zhu, Tianqing Zhou, Wanlei Han, Xiaotong |
| author_facet | Gao, Chen Liu, Chi Luo, Zhengquan Xiao, Dongfu Sui, Maiying Shen, Sheng Zhu, Congcong Chen, Huajie Zuo, Xuhan Ge, Zongyuan Zhu, Tianqing Zhou, Wanlei Han, Xiaotong |
| contents | Large language models (LLMs) are increasingly applied in computer science education for tasks such as tutoring, content generation, and code assessment. However, systematic evaluations aligned with formal curricula and certification standards remain limited. This study benchmarked four recent models, including GPT-5, DeepSeek-R1, Qwen-Plus, and Llama-3.3-70B-Instruct, using a dataset of 1,068 questions derived from six certification exams covering networking, office applications, and Java programming.
We evaluated performance across language (Chinese vs. English), cognitive levels based on Bloom's Taxonomy, domain knowledge, confidence-accuracy alignment, and robustness to input masking. Results showed that GPT-5 performed best on English-language certifications, while Qwen-Plus performed better in Chinese contexts. DeepSeek-R1 achieved the most balanced cross-lingual performance, whereas Llama-3.3 showed clear limitations in higher-order reasoning and robustness. All models performed worse on more complex tasks.
These findings provide empirical support for the integration of LLMs into computer science education and offer practical implications for curriculum design and assessment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_06898 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Are LLMs Ready for Computer Science Education? A Cross-Domain, Cross-Lingual and Cognitive-Level Evaluation Using Professional Certification Exams Gao, Chen Liu, Chi Luo, Zhengquan Xiao, Dongfu Sui, Maiying Shen, Sheng Zhu, Congcong Chen, Huajie Zuo, Xuhan Ge, Zongyuan Zhu, Tianqing Zhou, Wanlei Han, Xiaotong Computers and Society Large language models (LLMs) are increasingly applied in computer science education for tasks such as tutoring, content generation, and code assessment. However, systematic evaluations aligned with formal curricula and certification standards remain limited. This study benchmarked four recent models, including GPT-5, DeepSeek-R1, Qwen-Plus, and Llama-3.3-70B-Instruct, using a dataset of 1,068 questions derived from six certification exams covering networking, office applications, and Java programming. We evaluated performance across language (Chinese vs. English), cognitive levels based on Bloom's Taxonomy, domain knowledge, confidence-accuracy alignment, and robustness to input masking. Results showed that GPT-5 performed best on English-language certifications, while Qwen-Plus performed better in Chinese contexts. DeepSeek-R1 achieved the most balanced cross-lingual performance, whereas Llama-3.3 showed clear limitations in higher-order reasoning and robustness. All models performed worse on more complex tasks. These findings provide empirical support for the integration of LLMs into computer science education and offer practical implications for curriculum design and assessment. |
| title | Are LLMs Ready for Computer Science Education? A Cross-Domain, Cross-Lingual and Cognitive-Level Evaluation Using Professional Certification Exams |
| topic | Computers and Society |
| url | https://arxiv.org/abs/2604.06898 |