Image de couverture de livre

NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Prompts

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Zhang, Shudan, Zhao, Hanlin, Liu, Xiao, Zheng, Qinkai, Qi, Zehan, Gu, Xiaotao, Zhang, Xiaohan, Dong, Yuxiao, Tang, Jie
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language Machine Learning Software Engineering
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
par: Zheng, Qinkai, et autres
Publié: (2023)

HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization
par: Peng, Qiwei, et autres
Publié: (2024)

HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation
par: Yu, Zhaojian, et autres
Publié: (2024)

How Generation Architecture Shapes Code Complexity in Multi-Agent LLM Systems: A Paired Study on HumanEval
par: Ashrafi, Nazmus
Publié: (2026)

Addressing Data Leakage in HumanEval Using Combinatorial Test Design
par: Bradbury, Jeremy S., et autres
Publié: (2024)

HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agent
par: Wu, Jie JW, et autres
Publié: (2024)

EvoCodeBench: A Human-Performance Benchmark for Self-Evolving LLM-Driven Coding Systems
par: Zhang, Wentao, et autres
Publié: (2026)

Advancing GenAI Assisted Programming--A Comparative Study on Prompt Efficiency and Code Quality Between GPT-4 and GLM-4
par: Yang, Angus, et autres
Publié: (2024)

Dependency-Aware Code Naturalness
par: Yang, Chen, et autres
Publié: (2024)

SysTradeBench: An Iterative Build-Test-Patch Benchmark for Strategy-to-Code Trading Systems with Drift-Aware Diagnostics
par: Cao, Yuchen, et autres
Publié: (2026)

CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models
par: Yu, Hao, et autres
Publié: (2023)

ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code
par: Feng, Jia, et autres
Publié: (2024)

EffiBench: Benchmarking the Efficiency of Automatically Generated Code
par: Huang, Dong, et autres
Publié: (2024)

Revisiting the Role of Natural Language Code Comments in Code Translation
par: Gupta, Monika, et autres
Publié: (2026)

DSCodeBench: A Realistic Benchmark for Data Science Code Generation
par: Ouyang, Shuyin, et autres
Publié: (2025)

Do Not Treat Code as Natural Language: Implications for Repository-Level Code Generation and Beyond
par: Le-Anh, Minh, et autres
Publié: (2026)

NaturalEdit: Code Modification through Direct Interaction with Adaptive Natural Language Representation
par: Tang, Ningzhi, et autres
Publié: (2025)

PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization
par: Jing, Huihao, et autres
Publié: (2026)

ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
par: Xie, Bang, et autres
Publié: (2026)

ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generation
par: Chen, Yeheng, et autres
Publié: (2026)

CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval
par: Geng, Jiahui, et autres
Publié: (2026)

Prompt-based Code Completion via Multi-Retrieval Augmented Generation
par: Tan, Hanzhuo, et autres
Publié: (2024)

Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Models
par: Vishwakarma, Sanjay, et autres
Publié: (2024)

Zero-Shot Code Representation Learning via Prompt Tuning
par: Cui, Nan, et autres
Publié: (2024)

CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning
par: Lam, Man Ho, et autres
Publié: (2025)

CodeFuse-CommitEval: Towards Benchmarking LLM's Power on Commit Message and Code Change Inconsistency Detection
par: Zhang, Qingyu, et autres
Publié: (2025)

Top General Performance = Top Domain Performance? DomainCodeBench: A Multi-domain Code Generation Benchmark
par: Zheng, Dewu, et autres
Publié: (2024)

CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding
par: Luo, Hanjun, et autres
Publié: (2025)

Modularization is Better: Effective Code Generation with Modular Prompting
par: Pan, Ruwei, et autres
Publié: (2025)

WebVIA: A Web-based Vision-Language Agentic Framework for Interactive and Verifiable UI-to-Code Generation
par: Xu, Mingde, et autres
Publié: (2025)

Collu-Bench: A Benchmark for Predicting Language Model Hallucinations in Code
par: Jiang, Nan, et autres
Publié: (2024)

AdaptEval: A Benchmark for Evaluating Large Language Models on Code Snippet Adaptation
par: Zhang, Tanghaoran, et autres
Publié: (2026)

AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators
par: Chou, Jason, et autres
Publié: (2025)

CodeS: Natural Language to Code Repository via Multi-Layer Sketch
par: Zan, Daoguang, et autres
Publié: (2024)

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
par: Jain, Naman, et autres
Publié: (2024)

ClassEval-T: Evaluating Large Language Models in Class-Level Code Translation
par: Xue, Pengyu, et autres
Publié: (2024)

CLAP: Learning Transferable Binary Code Representations with Natural Language Supervision
par: Wang, Hao, et autres
Publié: (2024)

EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories
par: Li, Jia, et autres
Publié: (2024)

DevEval: Evaluating Code Generation in Practical Software Projects
par: Li, Jia, et autres
Publié: (2024)

EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations
par: Li, Jia, et autres
Publié: (2024)