Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.03878 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912807717437440 |
|---|---|
| author | Rosa, Giovanni Moreno-Lumbreras, David Robles, Gregorio González-Barahona, Jesús M. |
| author_facet | Rosa, Giovanni Moreno-Lumbreras, David Robles, Gregorio González-Barahona, Jesús M. |
| contents | Large Language Models (LLMs) are increasingly integrated into software development workflows, yet their behavior in structured, specification-driven processes remains poorly understood. This paper presents an empirical study design using CURRANTE, a Visual Studio Code extension that enables a human-in-the-loop workflow for LLM-assisted code generation. The tool guides developers through three sequential stages--Specification, Tests, and Function--allowing them to define requirements, generate and refine test suites, and produce functions that satisfy those tests. Participants will solve medium-difficulty problems from the LiveCodeBench dataset, while the tool records fine-grained interaction logs, effectiveness metrics (e.g., pass rate, all-pass completion), efficiency indicators (e.g., time-to-pass), and iteration behaviors. The study aims to analyze how human intervention in specification and test refinement influences the quality and dynamics of LLM-generated code. The results will provide empirical insights into the design of next-generation development environments that align human reasoning with model-driven code generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_03878 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Understanding Specification-Driven Code Generation with LLMs: An Empirical Study Design Rosa, Giovanni Moreno-Lumbreras, David Robles, Gregorio González-Barahona, Jesús M. Software Engineering Large Language Models (LLMs) are increasingly integrated into software development workflows, yet their behavior in structured, specification-driven processes remains poorly understood. This paper presents an empirical study design using CURRANTE, a Visual Studio Code extension that enables a human-in-the-loop workflow for LLM-assisted code generation. The tool guides developers through three sequential stages--Specification, Tests, and Function--allowing them to define requirements, generate and refine test suites, and produce functions that satisfy those tests. Participants will solve medium-difficulty problems from the LiveCodeBench dataset, while the tool records fine-grained interaction logs, effectiveness metrics (e.g., pass rate, all-pass completion), efficiency indicators (e.g., time-to-pass), and iteration behaviors. The study aims to analyze how human intervention in specification and test refinement influences the quality and dynamics of LLM-generated code. The results will provide empirical insights into the design of next-generation development environments that align human reasoning with model-driven code generation. |
| title | Understanding Specification-Driven Code Generation with LLMs: An Empirical Study Design |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2601.03878 |