Cover Image

SwiftEval: Developing a Language-Specific Benchmark for LLM-generated Code Evaluation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Petrukha, Ivan, Kurliak, Yana, Stulova, Nataliia
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Computation and Language Programming Languages Software Engineering
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM
by: Xia, Chunqiu Steven, et al.
Published: (2024)

SmartEval: A Benchmark for Evaluating LLM-Generated Smart Contracts from Natural Language Specifications
by: Goel, Abhinav, et al.
Published: (2026)

StackEval: Benchmarking LLMs in Coding Assistance
by: Shah, Nidhish, et al.
Published: (2024)

EffiPair: Improving the Efficiency of LLM-generated Code with Relative Contrastive Feedback
by: Hajizadeh, Samira, et al.
Published: (2026)

HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization
by: Peng, Qiwei, et al.
Published: (2024)

NExT: Teaching Large Language Models to Reason about Code Execution
by: Ni, Ansong, et al.
Published: (2024)

Language Models for Code Completion: A Practical Evaluation
by: Izadi, Maliheh, et al.
Published: (2024)

Grounding Data Science Code Generation with Input-Output Specifications
by: Wen, Yeming, et al.
Published: (2024)

Neural Models for Source Code Synthesis and Completion
by: Niyogi, Mitodru
Published: (2024)

SIMCOPILOT: Evaluating Large Language Models for Copilot-Style Code Generation
by: Jiang, Mingchao, et al.
Published: (2025)

A Multi-Perspective Architecture for Semantic Code Search
by: Haldar, Rajarshi, et al.
Published: (2020)

Anka: A Domain-Specific Language for Reliable LLM Code Generation
by: Mazrouei, Saif Khalfan Saif Al
Published: (2025)

VeriSoftBench: Repository-Scale Formal Verification Benchmarks for Lean
by: Xin, Yutong, et al.
Published: (2026)

TokDrift: When LLM Speaks in Subwords but Code Speaks in Grammar
by: Li, Yinxi, et al.
Published: (2025)

MonoCoder: Domain-Specific Code Language Model for HPC Codes and Tasks
by: Kadosh, Tal, et al.
Published: (2023)

JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language Models
by: Cao, Jialun, et al.
Published: (2024)

NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Prompts
by: Zhang, Shudan, et al.
Published: (2024)

Large Language Models for Multilingual Code Intelligence: A Survey
by: Jiang, Chao, et al.
Published: (2026)

CodeJudge: Evaluating Code Generation with Large Language Models
by: Tong, Weixi, et al.
Published: (2024)

EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checking
by: Wei, Anjiang, et al.
Published: (2025)

CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences
by: Weyssow, Martin, et al.
Published: (2024)

Evaluating Language Models for Efficient Code Generation
by: Liu, Jiawei, et al.
Published: (2024)

Benchmarking LLM Code Generation for Audio Programming with Visual Dataflow Languages
by: Zhang, William, et al.
Published: (2024)

Functional Consistency of LLM Code Embeddings: A Self-Evolving Data Synthesis Framework for Benchmarking
by: Li, Zhuohao, et al.
Published: (2025)

MLCPD: A Unified Multi-Language Code Parsing Dataset with Universal AST Schema
by: Gajjar, Jugal, et al.
Published: (2025)

CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation
by: Peng, Jinjun, et al.
Published: (2025)

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
by: Jain, Naman, et al.
Published: (2024)

Constrained Decoding for Fill-in-the-Middle Code Language Models via Efficient Left and Right Quotienting of Context-Sensitive Grammars
by: Melcer, Daniel, et al.
Published: (2024)

Evaluating the Generalization Capabilities of Large Language Models on Code Reasoning
by: Yang, Rem, et al.
Published: (2025)

Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models
by: Riddell, Martin, et al.
Published: (2024)

Lita: Light Agent Uncovers the Agentic Coding Capabilities of LLMs
by: Dai, Hankun, et al.
Published: (2025)

SynCode: LLM Generation with Grammar Augmentation
by: Ugare, Shubham, et al.
Published: (2024)

Evaluating Quantized Large Language Models for Code Generation on Low-Resource Language Benchmarks
by: Nyamsuren, Enkhbold
Published: (2024)

Uncertainty Awareness of Large Language Models Under Code Distribution Shifts: A Benchmark Study
by: Li, Yufei, et al.
Published: (2024)

Benchmarking LLM for Code Smells Detection: OpenAI GPT-4.0 vs DeepSeek-V3
by: Sadik, Ahmed R., et al.
Published: (2025)

GrowthHacker: Automated Off-Policy Evaluation Optimization Using Code-Modifying LLM Agents
by: Wu, Jie JW, et al.
Published: (2025)

Linguacodus: A Synergistic Framework for Transformative Code Generation in Machine Learning Pipelines
by: Trofimova, Ekaterina, et al.
Published: (2024)

PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization
by: Jing, Huihao, et al.
Published: (2026)

CONCUR: Benchmarking LLMs for Concurrent Code Generation
by: Huang, Jue, et al.
Published: (2026)

Automatically Testing Functional Properties of Code Translation Models
by: Eniser, Hasan Ferit, et al.
Published: (2023)