Cover Image

Are Large Language Models Memorizing Bug Benchmarks?

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Ramos, Daniel, Mamede, Claudia, Jain, Kush, Canelas, Paulo, Gamboa, Catarina, Goues, Claire Le
Format:	Preprint
Published:	2024
Subjects:	Software Engineering Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

SPELL: Synthesis of Programmatic Edits using LLMs
by: Ramos, Daniel, et al.
Published: (2026)

Are Sparse Autoencoders Useful for Java Function Bug Detection?
by: Melo, Rui, et al.
Published: (2025)

TestForge: Feedback-Driven, Agentic Test Suite Generation
by: Jain, Kush, et al.
Published: (2025)

Benchmarking Large Language Models with Integer Sequence Generation Tasks
by: O'Malley, Daniel, et al.
Published: (2024)

GREPO: A Benchmark for Graph Neural Networks on Repository-Level Bug Localization
by: Wang, Juntong, et al.
Published: (2026)

Promise and Peril of Collaborative Code Generation Models: Balancing Effectiveness and Memorization
by: Chen, Zhi, et al.
Published: (2024)

MELT: Mining Effective Lightweight Transformations from Pull Requests
by: Ramos, Daniel, et al.
Published: (2023)

Evaluating Robustness of Large Language Models in Enterprise Applications: Benchmarks for Perturbation Consistency Across Formats and Languages
by: Bogavelli, Tara, et al.
Published: (2026)

Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks
by: Li, Yuangang, et al.
Published: (2026)

Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models
by: Dong, Yihong, et al.
Published: (2024)

On the Impact of Code Comments for Automated Bug-Fixing: An Empirical Study
by: Vitale, Antonio, et al.
Published: (2026)

An Empirical Evaluation of Locally Deployed LLMs for Bug Detection in Python Code
by: Vulićević, Jelena Ilić
Published: (2026)

Imitation Game: Reproducing Deep Learning Bugs Leveraging an Intelligent Agent
by: Shah, Mehil B, et al.
Published: (2025)

DrugPlayGround: Benchmarking Large Language Models and Embeddings for Drug Discovery
by: Liu, Tianyu, et al.
Published: (2026)

Security Vulnerability Detection with Multitask Self-Instructed Fine-Tuning of Large Language Models
by: Yang, Aidan Z. H., et al.
Published: (2024)

SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents
by: Mündler, Niels, et al.
Published: (2024)

CODEMENV: Benchmarking Large Language Models on Code Migration
by: Cheng, Keyuan, et al.
Published: (2025)

TriagerX: Dual Transformers for Bug Triaging Tasks with Content and Interaction Based Rankings
by: Mamun, Md Afif Al, et al.
Published: (2025)

scicode-lint: Detecting Methodology Bugs in Scientific Python Code with LLM-Generated Patterns
by: Samsonau, Sergey V.
Published: (2026)

Automatic Programming: Large Language Models and Beyond
by: Lyu, Michael R., et al.
Published: (2024)

Generating Streamlining Constraints with Large Language Models
by: Voboril, Florentina, et al.
Published: (2024)

PreciseBugCollector: Extensible, Executable and Precise Bug-fix Collection
by: Ye, He, et al.
Published: (2023)

The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations?
by: Gu, Alex, et al.
Published: (2024)

Toward Explaining Large Language Models in Software Engineering Tasks
by: Vitale, Antonio, et al.
Published: (2025)

SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable Large Language Models in Resolving Real-World Bugs
by: Pham, Minh V. T., et al.
Published: (2025)

Assessing the Impact of Code Changes on the Fault Localizability of Large Language Models
by: Haroon, Sabaat, et al.
Published: (2025)

Exploring the Integration of Large Language Models in Industrial Test Maintenance Processes
by: Liu, Jingxiong, et al.
Published: (2024)

ProToken: Token-Level Attribution for Federated Large Language Models
by: Gill, Waris, et al.
Published: (2026)

Effective Large Language Model Debugging with Best-first Tree Search
by: Song, Jialin, et al.
Published: (2024)

Automated Factual Benchmarking for In-Car Conversational Systems using Large Language Models
by: Giebisch, Rafael, et al.
Published: (2025)

EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages
by: Sharma, Aman, et al.
Published: (2026)

Consolidating TinyML Lifecycle with Large Language Models: Reality, Illusion, or Opportunity?
by: Wu, Guanghan, et al.
Published: (2025)

Zero-Shot Attribution for Large Language Models: A Distribution Testing Approach
by: Canonne, Clément L., et al.
Published: (2025)

AKD : Adversarial Knowledge Distillation For Large Language Models Alignment on Coding tasks
by: Oulkadda, Ilyas, et al.
Published: (2025)

CONSTRUCTA: Automating Commercial Construction Schedules in Fabrication Facilities with Large Language Models
by: Zhang, Yifan, et al.
Published: (2025)

Improving Examples in Web API Specifications using Iterated-Calls In-Context Learning
by: Jain, Kush, et al.
Published: (2025)

Large Language Models Should Ask Clarifying Questions to Increase Confidence in Generated Code
by: Wu, Jie JW
Published: (2023)

Assessing Large Language Models for Automated Feedback Generation in Learning Programming Problem Solving
by: Silva, Priscylla, et al.
Published: (2025)

Accuracy, Stability, and Repeated-Run Reliability of Large Language Models on Deterministic Programming Tasks
by: Zhou, Yongxi, et al.
Published: (2026)

A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language Models
by: Wu, Yixi, et al.
Published: (2024)