Imagen de Portada

Where Do LLMs Still Struggle? An In-Depth Analysis of Code Generation Benchmarks

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Sharifloo, Amir Molzam, Heydari, Maedeh, Kazerooni, Parsa, Maninger, Daniel, Mezini, Mira
Formato:	Preprint
Publicado:	2025
Materias:	Software Engineering Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Evaluating and Mitigating Errors in LLM-Generated Web API Integrations
por: Maninger, Daniel, et al.
Publicado: (2025)

Deep Graph-Language Fusion for Structure-Aware Code Generation
por: Tiftikci, Mert, et al.
Publicado: (2026)

Towards Trustworthy AI Software Development Assistance
por: Maninger, Daniel, et al.
Publicado: (2023)

Domain-Adaptable Reinforcement Learning for Code Generation with Dense Rewards
por: Jolfaei, Erfan Aghadavoodi, et al.
Publicado: (2026)

Integrating Symbolic Execution into the Fine-Tuning of Code-Generating LLMs
por: Sakharova, Marina, et al.
Publicado: (2025)

A Critical Study of What Code-LLMs (Do Not) Learn
por: Anand, Abhinav, et al.
Publicado: (2024)

OSS-Bench: Benchmark Generator for Coding LLMs
por: Jiang, Yuancheng, et al.
Publicado: (2025)

The Struggles of LLMs in Cross-lingual Code Clone Detection
por: Moumoula, Micheline Bénédicte, et al.
Publicado: (2024)

Beyond BLEU: A Semantic Evaluation Method for Code Translation
por: Näumann, Julius, et al.
Publicado: (2026)

CONCUR: Benchmarking LLMs for Concurrent Code Generation
por: Huang, Jue, et al.
Publicado: (2026)

CodeSSM: Towards State Space Models for Code Understanding
por: Verma, Shweta, et al.
Publicado: (2025)

Drawing Pandas: A Benchmark for LLMs in Generating Plotting Code
por: Galimzyanov, Timur, et al.
Publicado: (2024)

CodeIF: Benchmarking the Instruction-Following Capabilities of Large Language Models for Code Generation
por: Yan, Kaiwen, et al.
Publicado: (2025)

Enabling Global, Human-Centered Explanations for LLMs:From Tokens to Interpretable Code and Test Generation
por: Khati, Dipin, et al.
Publicado: (2025)

The Fault in our Stars: Quality Assessment of Code Generation Benchmarks
por: Siddiq, Mohammed Latif, et al.
Publicado: (2024)

StackEval: Benchmarking LLMs in Coding Assistance
por: Shah, Nidhish, et al.
Publicado: (2024)

Automating API Documentation with LLMs: A BERTopic Approach
por: Naghshzan, AmirHossein
Publicado: (2025)

AICD Bench: A Challenging Benchmark for AI-Generated Code Detection
por: Orel, Daniil, et al.
Publicado: (2026)

Prism: Dynamic and Flexible Benchmarking of LLMs Code Generation with Monte Carlo Tree Search
por: Majdinasab, Vahid, et al.
Publicado: (2025)

Leveraging LLMs for Legacy Code Modernization: Challenges and Opportunities for LLM-Generated Documentation
por: Diggs, Colin, et al.
Publicado: (2024)

Teaching Code Refactoring Using LLMs
por: Khairnar, Anshul, et al.
Publicado: (2025)

Towards Verified Code Reasoning by LLMs
por: Sistla, Meghana, et al.
Publicado: (2025)

Automating Code Adaptation for MLOps -- A Benchmarking Study on LLMs
por: Patel, Harsh, et al.
Publicado: (2024)

Operational Robustness of LLMs on Code Generation
por: Paul, Debalina Ghosh, et al.
Publicado: (2026)

Understanding Robustness of Model Editing in Code LLMs
por: Chhetri, Vinaik, et al.
Publicado: (2025)

Renaissance of Literate Programming in the Era of LLMs: Enhancing LLM-Based Code Generation in Large-Scale Projects
por: Zhang, Wuyang, et al.
Publicado: (2024)

PromSec: Prompt Optimization for Secure Generation of Functional Source Code with Large Language Models (LLMs)
por: Nazzal, Mahmoud, et al.
Publicado: (2024)

Assessing the Quality and Security of AI-Generated Code: A Quantitative Analysis
por: Sabra, Abbas, et al.
Publicado: (2025)

Unsupervised Evaluation of Code LLMs with Round-Trip Correctness
por: Allamanis, Miltiadis, et al.
Publicado: (2024)

CodeTaste: Can LLMs Generate Human-Level Code Refactorings?
por: Thillen, Alex, et al.
Publicado: (2026)

ThrowBench: Benchmarking LLMs by Predicting Runtime Exceptions
por: Prenner, Julian Aron, et al.
Publicado: (2025)

Mechanistic Interpretability of Code Correctness in LLMs via Sparse Autoencoders
por: Tahimic, Kriz, et al.
Publicado: (2025)

SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
por: Raghavendra, Mohit, et al.
Publicado: (2026)

Exploring Code Language Models for Automated HLS-based Hardware Generation: Benchmark, Infrastructure and Analysis
por: Gai, Jiahao, et al.
Publicado: (2025)

Protocode: Prototype-Driven Interpretability for Code Generation in LLMs
por: Bodla, Krishna Vamshi, et al.
Publicado: (2025)

TritonRL: Training LLMs to Think and Code Triton Without Cheating
por: Woo, Jiin, et al.
Publicado: (2025)

K-ASTRO: Structure-Aware Adaptation of LLMs for Code Vulnerability Detection
por: Zhang, Yifan, et al.
Publicado: (2022)

Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs
por: Haque, Mirazul, et al.
Publicado: (2025)

SemRep: Generative Code Representation Learning with Code Transformations
por: Li, Weichen, et al.
Publicado: (2026)

Towards a Small Language Model Lifecycle Framework
por: Miraghaei, Parsa, et al.
Publicado: (2025)