Cover Image

Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Li, Yuangang, Chen, Justin Tian Jin, Yu, Ethan, Hong, David, Ahmed, Iftekhar
Format:	Preprint
Published:	2026
Subjects:	Software Engineering Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

A Deep Dive Into Large Language Model Code Generation Mistakes: What and Why?
by: Chen, QiHong, et al.
Published: (2024)

Output Format Biases in the Evaluation of Large Language Models for Code Translation
by: Macedo, Marcos, et al.
Published: (2024)

Beyond Self-learned Attention: Mitigating Attention Bias in Transformer-based Models Using Attention Guidance
by: Gesi, Jiri, et al.
Published: (2024)

Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models
by: Zheng, Jiasheng, et al.
Published: (2024)

Beyond Functional Correctness: Investigating Coding Style Inconsistencies in Large Language Models
by: Wang, Yanlin, et al.
Published: (2024)

AdaptEval: A Benchmark for Evaluating Large Language Models on Code Snippet Adaptation
by: Zhang, Tanghaoran, et al.
Published: (2026)

Beyond Retrieval: A Multitask Benchmark and Model for Code Search
by: Xue, Siqiao, et al.
Published: (2026)

Task Abstention for Large Language Models in Code Generation
by: Zhou, Yanke, et al.
Published: (2026)

CoCo-Bench: A Comprehensive Code Benchmark For Multi-task Large Language Model Evaluation
by: Yin, Wenjing, et al.
Published: (2025)

CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Models
by: Padwal, Vedant
Published: (2026)

COMPASS: A Multi-Dimensional Benchmark for Evaluating Code Generation in Large Language Models
by: Meaden, James, et al.
Published: (2025)

Benchmarking Correctness and Security in Multi-Turn Code Generation
by: Rawal, Ruchit, et al.
Published: (2025)

CodeMind: Evaluating Large Language Models for Code Reasoning
by: Liu, Changshu, et al.
Published: (2024)

Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination
by: Chen, Simin, et al.
Published: (2025)

TaskEval: Assessing Difficulty of Code Generation Tasks for Large Language Models
by: Tambon, Florian, et al.
Published: (2024)

Advancing Language Models for Code-related Tasks
by: Tian, Zhao
Published: (2026)

An Execution-Verified Multi-Language Benchmark for Code Semantic Reasoning
by: Li, Yikun, et al.
Published: (2026)

CoRE: A Fine-Grained Code Reasoning Benchmark Beyond Output Prediction
by: Gao, Jun, et al.
Published: (2026)

Squeez: Task-Conditioned Tool-Output Pruning for Coding Agents
by: Kovács, Ádám
Published: (2026)

Evaluating Large Language Models for Code Review
by: Cihan, Umut, et al.
Published: (2025)

SimdBench: Benchmarking Large Language Models for SIMD-Intrinsic Code Generation
by: He, Yibo, et al.
Published: (2025)

GeoCode-GPT: A Large Language Model for Geospatial Code Generation Tasks
by: Hou, Shuyang, et al.
Published: (2024)

Beyond Accuracy: Characterizing Code Comprehension Capabilities in (Large) Language Models
by: Mächtle, Felix, et al.
Published: (2026)

How Propense Are Large Language Models at Producing Code Smells? A Benchmarking Study
by: Velasco, Alejandro, et al.
Published: (2024)

Exploring and Unleashing the Power of Large Language Models in Automated Code Translation
by: Yang, Zhen, et al.
Published: (2024)

Cross-Task Benchmarking and Evaluation of General-Purpose and Code-Specific Large Language Models
by: Das, Gunjan, et al.
Published: (2025)

From Bias To Improved Prompts: A Case Study of Bias Mitigation of Clone Detection Models
by: Chen, QiHong, et al.
Published: (2025)

Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forward
by: Sergeyuk, Agnia, et al.
Published: (2024)

CoRe: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks
by: Xie, Danning, et al.
Published: (2025)

Automated Code Review Using Large Language Models with Symbolic Reasoning
by: Icoz, Busra, et al.
Published: (2025)

CRPE: Expanding The Reasoning Capability of Large Language Model for Code Generation
by: Gui, Ningxin, et al.
Published: (2025)

Beyond Summaries: Structure-Aware Labeling of Code Changes with Large Language Models
by: Weiss, Bar, et al.
Published: (2026)

Beyond Autoregression: An Empirical Study of Diffusion Large Language Models for Code Generation
by: Li, Chengze, et al.
Published: (2025)

Selecting and Combining Large Language Models for Scalable Code Clone Detection
by: Chochlov, Muslim, et al.
Published: (2025)

Towards Advancing Code Generation with Large Language Models: A Research Roadmap
by: Jin, Haolin, et al.
Published: (2025)

RePaCA: Leveraging Reasoning Large Language Models for Static Automated Patch Correctness Assessment
by: Fuster-Pena, Marcos, et al.
Published: (2025)

Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution
by: Shastry, KN Ajay, et al.
Published: (2026)

A Code Comprehension Benchmark for Large Language Models for Code
by: Havare, Jayant, et al.
Published: (2025)

Benchmarks and Metrics for Evaluations of Code Generation: A Critical Review
by: Paul, Debalina Ghosh, et al.
Published: (2024)

CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluation
by: Yang, Guang, et al.
Published: (2025)