Testing with AI Agents: An Empirical Study of Test Generation Frequency, Quality, and Coverage
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917341396205568 |
|---|---|
| author | Yoshimoto, Suzuka Fujita, Shun Horikawa, Kosei Feitosa, Daniel Kashiwa, Yutaro Iida, Hajimu |
| author_facet | Yoshimoto, Suzuka Fujita, Shun Horikawa, Kosei Feitosa, Daniel Kashiwa, Yutaro Iida, Hajimu |
| contents | Agent-based coding tools have transformed software development practices. Unlike prompt-based approaches that require developers to manually integrate generated code, these agent-based tools autonomously interact with repositories to create, modify, and execute code, including test generation. While many developers have adopted agent-based coding tools, little is known about how these tools generate tests in real-world development scenarios or how AI-generated tests compare to human-written ones.
This study presents an empirical analysis of test generation by agent-based coding tools using the AIDev dataset. We extracted 2,232 commits containing test-related changes and investigated three aspects: the frequency of test additions, the structural characteristics of the generated tests, and their impact on code coverage. Our findings reveal that (i) AI authored 16.4% of all commits adding tests in real-world repositories, (ii) AI-generated test methods exhibit distinct structural patterns, featuring longer code and a higher density of assertions while maintaining lower cyclomatic complexity through linear logic, and (iii) AI-generated tests contribute to code coverage comparable to human-written tests, frequently achieving positive coverage gains across several projects. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_13724 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Testing with AI Agents: An Empirical Study of Test Generation Frequency, Quality, and Coverage Yoshimoto, Suzuka Fujita, Shun Horikawa, Kosei Feitosa, Daniel Kashiwa, Yutaro Iida, Hajimu Software Engineering Agent-based coding tools have transformed software development practices. Unlike prompt-based approaches that require developers to manually integrate generated code, these agent-based tools autonomously interact with repositories to create, modify, and execute code, including test generation. While many developers have adopted agent-based coding tools, little is known about how these tools generate tests in real-world development scenarios or how AI-generated tests compare to human-written ones. This study presents an empirical analysis of test generation by agent-based coding tools using the AIDev dataset. We extracted 2,232 commits containing test-related changes and investigated three aspects: the frequency of test additions, the structural characteristics of the generated tests, and their impact on code coverage. Our findings reveal that (i) AI authored 16.4% of all commits adding tests in real-world repositories, (ii) AI-generated test methods exhibit distinct structural patterns, featuring longer code and a higher density of assertions while maintaining lower cyclomatic complexity through linear logic, and (iii) AI-generated tests contribute to code coverage comparable to human-written tests, frequently achieving positive coverage gains across several projects. |
| title | Testing with AI Agents: An Empirical Study of Test Generation Frequency, Quality, and Coverage |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2603.13724 |