Cover Image

Mutation-based Consistency Testing for Evaluating the Code Understanding Capability of LLMs

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Li, Ziyu, Shin, Donghwan
Format:	Preprint
Published:	2024
Subjects:	Software Engineering Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Code-Vision: Evaluating Multimodal LLMs Logic Understanding and Code Generation Capabilities
by: Wang, Hanbin, et al.
Published: (2025)

TREAT: A Code LLMs Trustworthiness / Reliability Evaluation and Testing Framework
by: Gao, Shuzheng, et al.
Published: (2025)

Do Code LLMs Understand Design Patterns?
by: Pan, Zhenyu, et al.
Published: (2025)

Flow2Code: Evaluating Large Language Models for Flowchart-based Code Generation Capability
by: He, Mengliang, et al.
Published: (2025)

Are Large Language Models Robust in Understanding Code Against Semantics-Preserving Mutations?
by: Orvalho, Pedro, et al.
Published: (2025)

Re-Evaluating Code LLM Benchmarks Under Semantic Mutation
by: Pan, Zhiyuan, et al.
Published: (2025)

Domain Adaptation for Code Model-based Unit Test Case Generation
by: Shin, Jiho, et al.
Published: (2023)

CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
by: Yan, Weixiang, et al.
Published: (2023)

An Empirical Study on Capability of Large Language Models in Understanding Code Semantics
by: Nguyen, Thu-Trang, et al.
Published: (2024)

An Insight into Security Code Review with LLMs: Capabilities, Obstacles, and Influential Factors
by: Yu, Jiaxin, et al.
Published: (2024)

The Code Barrier: What LLMs Actually Understand?
by: Nikiema, Serge Lionel, et al.
Published: (2025)

CoRe: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks
by: Xie, Danning, et al.
Published: (2025)

Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion?
by: Pan, Zhenyu, et al.
Published: (2024)

CodeWatcher: IDE Telemetry Data Extraction Tool for Understanding Coding Interactions with LLMs
by: Basha, Manaal, et al.
Published: (2025)

Unify and Triumph: Polyglot, Diverse, and Self-Consistent Generation of Unit Tests with LLMs
by: Khelladi, Djamel Eddine, et al.
Published: (2025)

ReCatcher: Towards LLMs Regression Testing for Code Generation
by: Abbassi, Altaf Allah, et al.
Published: (2025)

Evaluating the Energy-Efficiency of the Code Generated by LLMs
by: Islam, Md Arman, et al.
Published: (2025)

Assessing Code Understanding in LLMs
by: Laneve, Cosimo, et al.
Published: (2025)

MutaBot: A Mutation Testing Approach for Chatbots
by: Urrico, Michael Ferdinando, et al.
Published: (2024)

Holistic Evaluation of State-of-the-Art LLMs for Code Generation
by: Zhang, Le, et al.
Published: (2025)

CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Models
by: Padwal, Vedant
Published: (2026)

Understanding the Limits of Automated Evaluation for Code Review Bots in Practice
by: Karakaya, Veli, et al.
Published: (2026)

FastCode: Fast and Cost-Efficient Code Understanding and Reasoning
by: Li, Zhonghang, et al.
Published: (2026)

Coding in a Bubble? Evaluating LLMs in Resolving Context Adaptation Bugs During Code Adaptation
by: Zhang, Tanghaoran, et al.
Published: (2026)

Mutation-Guided LLM-based Test Generation at Meta
by: Foster, Christopher, et al.
Published: (2025)

TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenance
by: Bruches, Elena, et al.
Published: (2026)

Helping LLMs Improve Code Generation Using Feedback from Testing and Static Analysis
by: Dolcetti, Greta, et al.
Published: (2024)

CodeIF-Bench: Evaluating Instruction-Following Capabilities of Large Language Models in Interactive Code Generation
by: Wang, Peiding, et al.
Published: (2025)

Bias Testing and Mitigation in LLM-based Code Generation
by: Huang, Dong, et al.
Published: (2023)

Evaluation of the Code Generation Capabilities of ChatGPT 4: A Comparative Analysis in 19 Programming Languages
by: Gilbert, L. C.
Published: (2025)

QuanForge: A Mutation Testing Framework for Quantum Neural Networks
by: Shao, Minqi, et al.
Published: (2026)

Open the Oyster: Empirical Evaluation and Improvement of Code Reasoning Confidence in LLMs
by: Wang, Shufan, et al.
Published: (2025)

Adaptive Testing for LLM-Based Applications: A Diversity-based Approach
by: Yoon, Juyeon, et al.
Published: (2025)

R-Log: Incentivizing Log Analysis Capability in LLMs via Reasoning-based Reinforcement Learning
by: Liu, Yilun, et al.
Published: (2025)

MIST-RL: Mutation-based Incremental Suite Testing via Reinforcement Learning
by: Zhu, Sicheng, et al.
Published: (2026)

Assessing the Code Clone Detection Capability of Large Language Models
by: Zhang, Zixian, et al.
Published: (2024)

Klear-CodeTest: Scalable Test Case Generation for Code Reinforcement Learning
by: Fu, Jia, et al.
Published: (2025)

Can We Make Code Green? Understanding Trade-Offs in LLMs vs. Human Code Optimizations
by: Rani, Pooja, et al.
Published: (2025)

Function-to-Style Guidance of LLMs for Code Translation
by: Zhang, Longhui, et al.
Published: (2025)

What Makes Code Generation Ethically Sourced?
by: Xu, Zhuolin, et al.
Published: (2025)