Cover Image

Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Riddell, Martin, Ni, Ansong, Cohan, Arman
Format:	Preprint
Published:	2024
Subjects:	Software Engineering Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

NExT: Teaching Large Language Models to Reason about Code Execution
by: Ni, Ansong, et al.
Published: (2024)

Evaluating the Generalization Capabilities of Large Language Models on Code Reasoning
by: Yang, Rem, et al.
Published: (2025)

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
by: Jain, Naman, et al.
Published: (2024)

HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation
by: Yu, Zhaojian, et al.
Published: (2024)

CodeJudge: Evaluating Code Generation with Large Language Models
by: Tong, Weixi, et al.
Published: (2024)

Evaluating Language Models for Efficient Code Generation
by: Liu, Jiawei, et al.
Published: (2024)

CodeEditorBench: Evaluating Code Editing Capability of Large Language Models
by: Guo, Jiawei, et al.
Published: (2024)

TAROT: Test-driven and Capability-adaptive Curriculum Reinforcement Fine-tuning for Code Generation with Large Language Models
by: Park, Chansung, et al.
Published: (2026)

Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models
by: Weyssow, Martin, et al.
Published: (2023)

CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences
by: Weyssow, Martin, et al.
Published: (2024)

Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models
by: Dong, Yihong, et al.
Published: (2024)

CodeVisionary: An Agent-based Framework for Evaluating Large Language Models in Code Generation
by: Wang, Xinchen, et al.
Published: (2025)

Prompting and Fine-tuning Large Language Models for Automated Code Review Comment Generation
by: Haider, Md. Asif, et al.
Published: (2024)

Neuron Patching: Semantic-based Neuron-level Language Model Repair for Code Generation
by: Gu, Jian, et al.
Published: (2023)

GLLM: Self-Corrective G-Code Generation using Large Language Models with User Feedback
by: Abdelaal, Mohamed, et al.
Published: (2025)

CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation
by: Peng, Jinjun, et al.
Published: (2025)

Towards Understanding What Code Language Models Learned
by: Ahmed, Toufique, et al.
Published: (2023)

Evaluating the Formal Reasoning Capabilities of Large Language Models through Chomsky Hierarchy
by: Dong, Yihong, et al.
Published: (2026)

Assessing the Latent Automated Program Repair Capabilities of Large Language Models using Round-Trip Translation
by: Ruiz, Fernando Vallecillos, et al.
Published: (2024)

Can OpenSource beat ChatGPT? -- A Comparative Study of Large Language Models for Text-to-Code Generation
by: Mayer, Luis, et al.
Published: (2024)

ReflexiCoder: Teaching Large Language Models to Self-Reflect on Generated Code and Self-Correct It via Reinforcement Learning
by: Jiang, Juyong, et al.
Published: (2026)

SIMCOPILOT: Evaluating Large Language Models for Copilot-Style Code Generation
by: Jiang, Mingchao, et al.
Published: (2025)

SelfCodeAlign: Self-Alignment for Code Generation
by: Wei, Yuxiang, et al.
Published: (2024)

IntentCoding: Amplifying User Intent in Code Generation
by: Fang, Zheng, et al.
Published: (2026)

Test Code Generation for Telecom Software Systems using Two-Stage Generative Model
by: Nabeel, Mohamad, et al.
Published: (2024)

Suggesting Code Edits in Interactive Machine Learning Notebooks Using Large Language Models
by: Jin, Bihui, et al.
Published: (2025)

LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs
by: Xia, Yunhui, et al.
Published: (2025)

Evaluation and Improvement of Fault Detection for Large Language Models
by: Hu, Qiang, et al.
Published: (2024)

RepoQA: Evaluating Long Context Code Understanding
by: Liu, Jiawei, et al.
Published: (2024)

SwiftEval: Developing a Language-Specific Benchmark for LLM-generated Code Evaluation
by: Petrukha, Ivan, et al.
Published: (2025)

DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
by: Guo, Daya, et al.
Published: (2024)

Uncertainty Awareness of Large Language Models Under Code Distribution Shifts: A Benchmark Study
by: Li, Yufei, et al.
Published: (2024)

Evaluating the Process Modeling Abilities of Large Language Models -- Preliminary Foundations and Results
by: Fettke, Peter, et al.
Published: (2025)

CONCUR: Benchmarking LLMs for Concurrent Code Generation
by: Huang, Jue, et al.
Published: (2026)

Language Models for Code Completion: A Practical Evaluation
by: Izadi, Maliheh, et al.
Published: (2024)

Wisdom and Delusion of LLM Ensembles for Code Generation and Repair
by: Vallecillos-Ruiz, Fernando, et al.
Published: (2025)

GiFT: Gibbs Fine-Tuning for Code Generation
by: Li, Haochen, et al.
Published: (2025)

MetaLint: Easy-to-Hard Generalization for Code Linting
by: Naik, Atharva, et al.
Published: (2025)

Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination
by: Chen, Simin, et al.
Published: (2025)

AST-T5: Structure-Aware Pretraining for Code Generation and Understanding
by: Gong, Linyuan, et al.
Published: (2024)