Cover Image

NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Singhal, Manav, Aggarwal, Tushar, Awasthi, Abhijeet, Natarajan, Nagarajan, Kanade, Aditya
Format:	Preprint
Published:	2024
Subjects:	Software Engineering Artificial Intelligence Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Robust Learning of Diverse Code Edits
by: Aggarwal, Tushar, et al.
Published: (2025)

Code Researcher: Deep Research Agent for Large Systems Code and Commit History
by: Singh, Ramneet, et al.
Published: (2025)

Legal Requirements Translation from Law
by: Singhal, Anmol, et al.
Published: (2025)

Requirements Elicitation Follow-Up Question Generation
by: Shen, Yuchen, et al.
Published: (2025)

MASAI: Modular Architecture for Software-engineering AI Agents
by: Arora, Daman, et al.
Published: (2024)

CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?
by: Zhao, Yuwei, et al.
Published: (2024)

DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories
by: Li, Jia, et al.
Published: (2024)

Is Your AI-Generated Code Really Safe? Evaluating Large Language Models on Secure Code Generation with CodeSecEval
by: Wang, Jiexin, et al.
Published: (2024)

Isolating Language-Coding from Problem-Solving: Benchmarking LLMs with PseudoEval
by: Wu, Jiarong, et al.
Published: (2025)

ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generation
by: Chen, Yeheng, et al.
Published: (2026)

IndicEval-XL: Bridging Linguistic Diversity in Code Generation Across Indic Languages
by: Singh, Ujjwal, et al.
Published: (2025)

M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluation
by: Liu, Jiaheng, et al.
Published: (2024)

MaintainCoder: Maintainable Code Generation Under Dynamic Requirements
by: Wang, Zhengren, et al.
Published: (2025)

StackEval: Benchmarking LLMs in Coding Assistance
by: Shah, Nidhish, et al.
Published: (2024)

ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generation
by: Liu, Kaiyuan, et al.
Published: (2025)

HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation
by: Yu, Zhaojian, et al.
Published: (2024)

SR-Eval: Evaluating LLMs on Code Generation under Stepwise Requirement Refinement
by: Zhan, Zexun, et al.
Published: (2025)

FunPRM: Function-as-Step Process Reward Model with Meta Reward Correction for Code Generation
by: Zhang, Ruiyi, et al.
Published: (2026)

Contract Based Verification of Non-functional Requirements for Embedded Automotive C Code
by: Amilon, Jesper, et al.
Published: (2026)

When the Code Autopilot Breaks: Why LLMs Falter in Embedded Machine Learning
by: Morabito, Roberto, et al.
Published: (2025)

Dealing with Data for RE: Mitigating Challenges while using NLP and Generative AI
by: Ghaisas, Smita, et al.
Published: (2024)

HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization
by: Peng, Qiwei, et al.
Published: (2024)

DevEval: Evaluating Code Generation in Practical Software Projects
by: Li, Jia, et al.
Published: (2024)

DARS: Dynamic Action Re-Sampling to Enhance Coding Agent Performance by Adaptive Tree Traversal
by: Aggarwal, Vaibhav, et al.
Published: (2025)

SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysis
by: Li, Yansong, et al.
Published: (2025)

Is Functional Correctness Enough to Evaluate Code Language Models? Exploring Diversity of Generated Codes
by: Chon, Heejae, et al.
Published: (2024)

Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models
by: Zheng, Jiasheng, et al.
Published: (2024)

Correctness Witnesses with Function Contracts
by: Heizmann, Matthias, et al.
Published: (2025)

ConCodeEval: Evaluating Large Language Models for Code Constraints in Domain-Specific Languages
by: Kammakomati, Mehant, et al.
Published: (2024)

NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Prompts
by: Zhang, Shudan, et al.
Published: (2024)

BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?
by: Chen, Guoxin, et al.
Published: (2026)

CODEPROMPTZIP: Code-specific Prompt Compression for Retrieval-Augmented Generation in Coding Tasks with LMs
by: He, Pengfei, et al.
Published: (2025)

ArchCode: Incorporating Software Requirements in Code Generation with Large Language Models
by: Han, Hojae, et al.
Published: (2024)

$\textbf{PLUM}$: Improving Code LMs with Execution-Guided On-Policy Preference Learning Driven By Synthetic Test Cases
by: Zhang, Dylan, et al.
Published: (2024)

Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM
by: Xia, Chunqiu Steven, et al.
Published: (2024)

Environment-Aware Code Generation: How far are We?
by: Wu, Tongtong, et al.
Published: (2026)

DependEval: Benchmarking LLMs for Repository Dependency Understanding
by: Du, Junjia, et al.
Published: (2025)

VHDL-Eval: A Framework for Evaluating Large Language Models in VHDL Code Generation
by: Vijayaraghavan, Prashanth, et al.
Published: (2024)

SwiftEval: Developing a Language-Specific Benchmark for LLM-generated Code Evaluation
by: Petrukha, Ivan, et al.
Published: (2025)

Vulnerability Detection with Code Language Models: How Far Are We?
by: Ding, Yangruibo, et al.
Published: (2024)