CodeSimpleQA: Scaling Factuality in Code Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917163875434496 |
|---|---|
| author | Yang, Jian Zhang, Wei Li, Yizhi Guo, Shawn Wang, Haowen Liu, Aishan Zhang, Ge Wang, Zili Li, Zhoujun Liu, Xianglong Lv, Weifeng |
| author_facet | Yang, Jian Zhang, Wei Li, Yizhi Guo, Shawn Wang, Haowen Liu, Aishan Zhang, Ge Wang, Zili Li, Zhoujun Liu, Xianglong Lv, Weifeng |
| contents | Large language models (LLMs) have made significant strides in code generation, achieving impressive capabilities in synthesizing code snippets from natural language instructions. However, a critical challenge remains in ensuring LLMs generate factually accurate responses about programming concepts, technical implementations, etc. Most previous code-related benchmarks focus on code execution correctness, overlooking the factual accuracy of programming knowledge. To address this gap, we present CodeSimpleQA, a comprehensive bilingual benchmark designed to evaluate the factual accuracy of code LLMs in answering code-related questions, which contains carefully curated question-answer pairs in both English and Chinese, covering diverse programming languages and major computer science domains. Further, we create CodeSimpleQA-Instruct, a large-scale instruction corpus with 66M samples, and develop a post-training framework combining supervised fine-tuning and reinforcement learning. Our comprehensive evaluation of diverse LLMs reveals that even frontier LLMs struggle with code factuality. Our proposed framework demonstrates substantial improvements over the base model, underscoring the critical importance of factuality-aware alignment in developing reliable code LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_19424 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CodeSimpleQA: Scaling Factuality in Code Large Language Models Yang, Jian Zhang, Wei Li, Yizhi Guo, Shawn Wang, Haowen Liu, Aishan Zhang, Ge Wang, Zili Li, Zhoujun Liu, Xianglong Lv, Weifeng Computation and Language Large language models (LLMs) have made significant strides in code generation, achieving impressive capabilities in synthesizing code snippets from natural language instructions. However, a critical challenge remains in ensuring LLMs generate factually accurate responses about programming concepts, technical implementations, etc. Most previous code-related benchmarks focus on code execution correctness, overlooking the factual accuracy of programming knowledge. To address this gap, we present CodeSimpleQA, a comprehensive bilingual benchmark designed to evaluate the factual accuracy of code LLMs in answering code-related questions, which contains carefully curated question-answer pairs in both English and Chinese, covering diverse programming languages and major computer science domains. Further, we create CodeSimpleQA-Instruct, a large-scale instruction corpus with 66M samples, and develop a post-training framework combining supervised fine-tuning and reinforcement learning. Our comprehensive evaluation of diverse LLMs reveals that even frontier LLMs struggle with code factuality. Our proposed framework demonstrates substantial improvements over the base model, underscoring the critical importance of factuality-aware alignment in developing reliable code LLMs. |
| title | CodeSimpleQA: Scaling Factuality in Code Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2512.19424 |