CoRE: A Fine-Grained Code Reasoning Benchmark Beyond Output Prediction
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911628166955008 |
|---|---|
| author | Gao, Jun Peng, Yun Qiao, Qian Zhou, Changhai Zhou, Yuhua Zhang, Shiyang Weng, Shichao Xing, Zhenchang Ren, Xiaoxue |
| author_facet | Gao, Jun Peng, Yun Qiao, Qian Zhou, Changhai Zhou, Yuhua Zhang, Shiyang Weng, Shichao Xing, Zhenchang Ren, Xiaoxue |
| contents | Despite strong performance on code generation tasks, it remains unclear whether large language models (LLMs) genuinely reason about code execution. Existing code reasoning benchmarks primarily evaluate final output correctness under a single canonical implementation, leaving two critical aspects underexplored: (1) whether LLMs can maintain consistency to functionally equivalent implementations, and (2) whether LLMs can accurately reason about intermediate execution states. We introduce \textbf{CoRE}, a \textbf{Co}de \textbf{Re}asoning benchmark that evaluates code reasoning through \textbf{implementation invariance} and \textbf{process transparency}. Extensive evaluations on eight frontier LLMs reveal two fundamental limitations. First, models exhibit a substantial \textbf{robustness gap}, with performance varying significantly across equivalent implementations. Second, we observe \textbf{superficial execution}, where models arrive at correct final outputs without correctly reasoning about intermediate execution states. Together, these findings demonstrate that output-only evaluations are insufficient for assessing code reasoning and position CoRE as a necessary benchmark for evaluating robust and faithful code reasoning.\footnote{Data and code are available at https://github.com/ZJUSig/CoRE.} |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_25399 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | CoRE: A Fine-Grained Code Reasoning Benchmark Beyond Output Prediction Gao, Jun Peng, Yun Qiao, Qian Zhou, Changhai Zhou, Yuhua Zhang, Shiyang Weng, Shichao Xing, Zhenchang Ren, Xiaoxue Software Engineering Despite strong performance on code generation tasks, it remains unclear whether large language models (LLMs) genuinely reason about code execution. Existing code reasoning benchmarks primarily evaluate final output correctness under a single canonical implementation, leaving two critical aspects underexplored: (1) whether LLMs can maintain consistency to functionally equivalent implementations, and (2) whether LLMs can accurately reason about intermediate execution states. We introduce \textbf{CoRE}, a \textbf{Co}de \textbf{Re}asoning benchmark that evaluates code reasoning through \textbf{implementation invariance} and \textbf{process transparency}. Extensive evaluations on eight frontier LLMs reveal two fundamental limitations. First, models exhibit a substantial \textbf{robustness gap}, with performance varying significantly across equivalent implementations. Second, we observe \textbf{superficial execution}, where models arrive at correct final outputs without correctly reasoning about intermediate execution states. Together, these findings demonstrate that output-only evaluations are insufficient for assessing code reasoning and position CoRE as a necessary benchmark for evaluating robust and faithful code reasoning.\footnote{Data and code are available at https://github.com/ZJUSig/CoRE.} |
| title | CoRE: A Fine-Grained Code Reasoning Benchmark Beyond Output Prediction |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2604.25399 |