Cover Image

Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Hu, Ruida, Wang, Xinchen, Peng, Chao, Gao, Cuiyun, Lo, David
Format:	Preprint
Published:	2026
Subjects:	Software Engineering Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Evaluating Repository-level Software Documentation via Question Answering and Feature-Driven Development
by: Wang, Xinchen, et al.
Published: (2026)

CodeVisionary: An Agent-based Framework for Evaluating Large Language Models in Code Generation
by: Wang, Xinchen, et al.
Published: (2025)

AEGIS: An Agent-based Framework for General Bug Reproduction from Issue Descriptions
by: Wang, Xinchen, et al.
Published: (2024)

CodeRepoQA: A Large-scale Benchmark for Software Engineering Question Answering
by: Hu, Ruida, et al.
Published: (2024)

Repo2Run: Automated Building Executable Environment for Code Repository at Scale
by: Hu, Ruida, et al.
Published: (2025)

Benchmarking LLMs for Fine-Grained Code Review with Enriched Context in Practice
by: Hu, Ruida, et al.
Published: (2025)

Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling
by: Trae Research Team, et al.
Published: (2025)

VulEval: Towards Repository-Level Evaluation of Software Vulnerability Detection
by: Wen, Xin-Cheng, et al.
Published: (2024)

EvoDev: An Iterative Feature-Driven Framework for End-to-End Software Development with LLM-based Agents
by: Liu, Junwei, et al.
Published: (2025)

AXIOM: Benchmarking LLM-as-a-Judge for Code via Rule-Based Perturbation and Multisource Quality Calibration
by: Wang, Ruiqi, et al.
Published: (2025)

Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents
by: Chen, Zhi, et al.
Published: (2026)

On the Replicability and Reproducibility of Deep Learning in Software Engineering
by: Liu, Chao, et al.
Published: (2020)

Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering
by: Wang, Ruiqi, et al.
Published: (2025)

RepoMasterEval: Evaluating Code Completion via Real-World Repositories
by: Wu, Qinyun, et al.
Published: (2024)

LLM-Based Test Case Generation in DBMS through Monte Carlo Tree Search
by: Chen, Yujia, et al.
Published: (2026)

EvolveTool-Bench: Evaluating the Quality of LLM-Generated Tool Libraries as Software Artifacts
by: Kaliyev, Alibek T., et al.
Published: (2026)

EvalSVA: Multi-Agent Evaluators for Next-Gen Software Vulnerability Assessment
by: Wen, Xin-Cheng, et al.
Published: (2024)

SR-Eval: Evaluating LLMs on Code Generation under Stepwise Requirement Refinement
by: Zhan, Zexun, et al.
Published: (2025)

A Real-World Benchmark for Evaluating Fine-Grained Issue Solving Capabilities of Large Language Models
by: Hu, Ruida, et al.
Published: (2024)

Evaluating LLM-Based Test Generation Under Software Evolution
by: Haroon, Sabaat, et al.
Published: (2026)

SWE-Fuse: Empowering Software Agents via Issue-free Trajectory Learning and Entropy-aware RLVR Training
by: Wen, Xin-Cheng, et al.
Published: (2026)

Argus: Resilience-Oriented Safety Assurance Framework for End-to-End ADSs
by: Wang, Dingji, et al.
Published: (2025)

Agents in the Sandbox: End-to-End Crash Bug Reproduction for Minecraft
by: Yapağcı, Eray, et al.
Published: (2025)

E2Edev: Benchmarking Large Language Models in End-to-End Software Development Task
by: Liu, Jingyao, et al.
Published: (2025)

AutoEmpirical: LLM-Based Automated Research for Empirical Software Fault Analysis
by: Yu, Jiongchi, et al.
Published: (2025)

LibRec: Benchmarking Retrieval-Augmented LLMs for Library Migration Recommendations
by: Han, Junxiao, et al.
Published: (2025)

ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development
by: Lu, Pengrui, et al.
Published: (2026)

Tree-of-Code: A Tree-Structured Exploring Framework for End-to-End Code Generation and Execution in Complex Task Handling
by: Ni, Ziyi, et al.
Published: (2024)

Vul-R2: A Reasoning LLM for Automated Vulnerability Repair
by: Wen, Xin-Cheng, et al.
Published: (2025)

DecompileBench: A Comprehensive Benchmark for Evaluating Decompilers in Real-World Scenarios
by: Gao, Zeyu, et al.
Published: (2025)

FrontendBench: A Benchmark for Evaluating LLMs on Front-End Development via Automatic Evaluation
by: Zhu, Hongda, et al.
Published: (2025)

MarsCode Agent: AI-native Automated Bug Fixing
by: Liu, Yizhou, et al.
Published: (2024)

The Prompt Alchemist: Automated LLM-Tailored Prompt Optimization for Test Case Generation
by: Gao, Shuzheng, et al.
Published: (2025)

Scalable Back-End for an AI-Based Diabetes Prediction Application
by: Radityo, Henry Anand Septian, et al.
Published: (2025)

ScenEval: A Benchmark for Scenario-Based Evaluation of Code Generation
by: Paul, Debalina Ghosh, et al.
Published: (2024)

Search-Based LLMs for Code Optimization
by: Gao, Shuzheng, et al.
Published: (2024)

Agentic Software Issue Resolution with Large Language Models: A Survey
by: Jiang, Zhonghao, et al.
Published: (2025)

Towards Specification-Driven LLM-Based Generation of Embedded Automotive Software
by: Patil, Minal Suresh, et al.
Published: (2024)

Copilot Evaluation Harness: Evaluating LLM-Guided Software Programming
by: Agarwal, Anisha, et al.
Published: (2024)

Towards Fair Machine Learning Software: Understanding and Addressing Model Bias Through Counterfactual Thinking
by: Wang, Zichong, et al.
Published: (2023)