TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenance
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908788769947648 |
|---|---|
| author | Bruches, Elena Alperovich, Vadim Baturova, Dari Derunets, Roman Grebenkin, Daniil Mkrtchyan, Georgy Sedukhin, Oleg Klementev, Mikhail Bondarenko, Ivan Bushkov, Nikolay Moiseev, Stanislav |
| author_facet | Bruches, Elena Alperovich, Vadim Baturova, Dari Derunets, Roman Grebenkin, Daniil Mkrtchyan, Georgy Sedukhin, Oleg Klementev, Mikhail Bondarenko, Ivan Bushkov, Nikolay Moiseev, Stanislav |
| contents | While Large Language Models (LLMs) have shown promise in software engineering, their application to unit testing remains largely confined to isolated test generation or oracle prediction, neglecting the broader challenge of test suite maintenance. We introduce TAM-Eval (Test Automated Maintenance Evaluation), a framework and benchmark designed to evaluate model performance across three core test maintenance scenarios: creation, repair, and updating of test suites. Unlike prior work limited to function-level tasks, TAM-Eval operates at the test file level, while maintaining access to full repository context during isolated evaluation, better reflecting real-world maintenance workflows. Our benchmark comprises 1,539 automatically extracted and validated scenarios from Python, Java, and Go projects. TAM-Eval supports system-agnostic evaluation of both raw LLMs and agentic workflows, using a reference-free protocol based on test suite pass rate, code coverage, and mutation testing. Empirical results indicate that state-of-the-art LLMs have limited capabilities in realistic test maintenance processes and yield only marginal improvements in test effectiveness. We release TAM-Eval as an open-source framework to support future research in automated software testing. Our data and code are publicly available at https://github.com/trndcenter/TAM-Eval. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_18241 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenance Bruches, Elena Alperovich, Vadim Baturova, Dari Derunets, Roman Grebenkin, Daniil Mkrtchyan, Georgy Sedukhin, Oleg Klementev, Mikhail Bondarenko, Ivan Bushkov, Nikolay Moiseev, Stanislav Software Engineering Artificial Intelligence While Large Language Models (LLMs) have shown promise in software engineering, their application to unit testing remains largely confined to isolated test generation or oracle prediction, neglecting the broader challenge of test suite maintenance. We introduce TAM-Eval (Test Automated Maintenance Evaluation), a framework and benchmark designed to evaluate model performance across three core test maintenance scenarios: creation, repair, and updating of test suites. Unlike prior work limited to function-level tasks, TAM-Eval operates at the test file level, while maintaining access to full repository context during isolated evaluation, better reflecting real-world maintenance workflows. Our benchmark comprises 1,539 automatically extracted and validated scenarios from Python, Java, and Go projects. TAM-Eval supports system-agnostic evaluation of both raw LLMs and agentic workflows, using a reference-free protocol based on test suite pass rate, code coverage, and mutation testing. Empirical results indicate that state-of-the-art LLMs have limited capabilities in realistic test maintenance processes and yield only marginal improvements in test effectiveness. We release TAM-Eval as an open-source framework to support future research in automated software testing. Our data and code are publicly available at https://github.com/trndcenter/TAM-Eval. |
| title | TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenance |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2601.18241 |