Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.04644 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912571564490752 |
|---|---|
| author | Boorlagadda, Subhang Atluri, Nitya Naga Sai Olmez, Muhammet Mustafa Gehringer, Edward F. |
| author_facet | Boorlagadda, Subhang Atluri, Nitya Naga Sai Olmez, Muhammet Mustafa Gehringer, Edward F. |
| contents | This paper explores the use of Large Language Models (LLMs) to automate the generation of test skeletons -- structural templates that outline unit test coverage without implementing full test logic. Test skeletons are especially important in test-driven development (TDD), where they provide an early framework for systematic verification. Traditionally authored manually, their creation can be time-consuming and error-prone, particularly in educational or large-scale development settings. We evaluate four LLMs -- GPT-4, DeepSeek-Chat, Llama4-Maverick, and Gemma2-9B -- on their ability to generate RSpec skeletons for a real-world Ruby class developed in a university software engineering course. Each model's output is assessed using static analysis and a blind expert review to measure structural correctness, clarity, maintainability, and conformance to testing best practices. The study reveals key differences in how models interpret code structure and testing conventions, offering insights into the practical challenges of using LLMs for automated test scaffolding. Our results show that DeepSeek generated the most maintainable and well-structured skeletons, while GPT-4 produced more complete but conventionally inconsistent output. The study reveals prompt design and contextual input as key quality factors. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_04644 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Comparative Evaluation of Large Language Models for Test-Skeleton Generation Boorlagadda, Subhang Atluri, Nitya Naga Sai Olmez, Muhammet Mustafa Gehringer, Edward F. Software Engineering This paper explores the use of Large Language Models (LLMs) to automate the generation of test skeletons -- structural templates that outline unit test coverage without implementing full test logic. Test skeletons are especially important in test-driven development (TDD), where they provide an early framework for systematic verification. Traditionally authored manually, their creation can be time-consuming and error-prone, particularly in educational or large-scale development settings. We evaluate four LLMs -- GPT-4, DeepSeek-Chat, Llama4-Maverick, and Gemma2-9B -- on their ability to generate RSpec skeletons for a real-world Ruby class developed in a university software engineering course. Each model's output is assessed using static analysis and a blind expert review to measure structural correctness, clarity, maintainability, and conformance to testing best practices. The study reveals key differences in how models interpret code structure and testing conventions, offering insights into the practical challenges of using LLMs for automated test scaffolding. Our results show that DeepSeek generated the most maintainable and well-structured skeletons, while GPT-4 produced more complete but conventionally inconsistent output. The study reveals prompt design and contextual input as key quality factors. |
| title | Comparative Evaluation of Large Language Models for Test-Skeleton Generation |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2509.04644 |