SecCodeBench-V2 Technical Report
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914336047366144 |
|---|---|
| author | Chen, Longfei Zhao, Ji Cui, Lanxiao Su, Tong Pan, Xingbo Li, Ziyang Wu, Yongxing Cao, Qijiang Cai, Qiyao Zhang, Jing Ni, Yuandong He, Junyao Zhang, Zeyu Ge, Chao Lu, Xuhuai Gao, Zeyu Cui, Yuxin Chen, Weisen Peng, Yuxuan Wang, Shengping Li, Qi Huang, Yukai Liu, Yukun Zhou, Tuo Zhuo, Terry Yue Lin, Junyang Zhang, Chao |
| author_facet | Chen, Longfei Zhao, Ji Cui, Lanxiao Su, Tong Pan, Xingbo Li, Ziyang Wu, Yongxing Cao, Qijiang Cai, Qiyao Zhang, Jing Ni, Yuandong He, Junyao Zhang, Zeyu Ge, Chao Lu, Xuhuai Gao, Zeyu Cui, Yuxin Chen, Weisen Peng, Yuxuan Wang, Shengping Li, Qi Huang, Yukai Liu, Yukun Zhou, Tuo Zhuo, Terry Yue Lin, Junyang Zhang, Chao |
| contents | We introduce SecCodeBench-V2, a publicly released benchmark for evaluating Large Language Model (LLM) copilots' capabilities of generating secure code. SecCodeBench-V2 comprises 98 generation and fix scenarios derived from Alibaba Group's industrial productions, where the underlying security issues span 22 common CWE (Common Weakness Enumeration) categories across five programming languages: Java, C, Python, Go, and JavaScript. SecCodeBench-V2 adopts a function-level task formulation: each scenario provides a complete project scaffold and requires the model to implement or patch a designated target function under fixed interfaces and dependencies. For each scenario, SecCodeBench-V2 provides executable proof-of-concept (PoC) test cases for both functional validation and security verification. All test cases are authored and double-reviewed by security experts, ensuring high fidelity, broad coverage, and reliable ground truth. Beyond the benchmark itself, we build a unified evaluation pipeline that assesses models primarily via dynamic execution. For most scenarios, we compile and run model-generated artifacts in isolated environments and execute PoC test cases to validate both functional correctness and security properties. For scenarios where security issues cannot be adjudicated with deterministic test cases, we additionally employ an LLM-as-a-judge oracle. To summarize performance across heterogeneous scenarios and difficulty levels, we design a Pass@K-based scoring protocol with principled aggregation over scenarios and severity, enabling holistic and comparable evaluation across models. Overall, SecCodeBench-V2 provides a rigorous and reproducible foundation for assessing the security posture of AI coding assistants, with results and artifacts released at https://alibaba.github.io/sec-code-bench. The benchmark is publicly available at https://github.com/alibaba/sec-code-bench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_15485 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SecCodeBench-V2 Technical Report Chen, Longfei Zhao, Ji Cui, Lanxiao Su, Tong Pan, Xingbo Li, Ziyang Wu, Yongxing Cao, Qijiang Cai, Qiyao Zhang, Jing Ni, Yuandong He, Junyao Zhang, Zeyu Ge, Chao Lu, Xuhuai Gao, Zeyu Cui, Yuxin Chen, Weisen Peng, Yuxuan Wang, Shengping Li, Qi Huang, Yukai Liu, Yukun Zhou, Tuo Zhuo, Terry Yue Lin, Junyang Zhang, Chao Cryptography and Security Artificial Intelligence Software Engineering We introduce SecCodeBench-V2, a publicly released benchmark for evaluating Large Language Model (LLM) copilots' capabilities of generating secure code. SecCodeBench-V2 comprises 98 generation and fix scenarios derived from Alibaba Group's industrial productions, where the underlying security issues span 22 common CWE (Common Weakness Enumeration) categories across five programming languages: Java, C, Python, Go, and JavaScript. SecCodeBench-V2 adopts a function-level task formulation: each scenario provides a complete project scaffold and requires the model to implement or patch a designated target function under fixed interfaces and dependencies. For each scenario, SecCodeBench-V2 provides executable proof-of-concept (PoC) test cases for both functional validation and security verification. All test cases are authored and double-reviewed by security experts, ensuring high fidelity, broad coverage, and reliable ground truth. Beyond the benchmark itself, we build a unified evaluation pipeline that assesses models primarily via dynamic execution. For most scenarios, we compile and run model-generated artifacts in isolated environments and execute PoC test cases to validate both functional correctness and security properties. For scenarios where security issues cannot be adjudicated with deterministic test cases, we additionally employ an LLM-as-a-judge oracle. To summarize performance across heterogeneous scenarios and difficulty levels, we design a Pass@K-based scoring protocol with principled aggregation over scenarios and severity, enabling holistic and comparable evaluation across models. Overall, SecCodeBench-V2 provides a rigorous and reproducible foundation for assessing the security posture of AI coding assistants, with results and artifacts released at https://alibaba.github.io/sec-code-bench. The benchmark is publicly available at https://github.com/alibaba/sec-code-bench. |
| title | SecCodeBench-V2 Technical Report |
| topic | Cryptography and Security Artificial Intelligence Software Engineering |
| url | https://arxiv.org/abs/2602.15485 |