Just-in-Time Catching Test Generation at Meta
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917234697306112 |
|---|---|
| author | Becker, Matthew Chen, Yifei Cochran, Nicholas Ghasemi, Pouyan Gulati, Abhishek Harman, Mark Haluza, Zachary Honarkhah, Mehrdad Robert, Herve Liu, Jiacheng Liu, Weini Thummala, Sreeja Yang, Xiaoning Xin, Rui Zeng, Sophie |
| author_facet | Becker, Matthew Chen, Yifei Cochran, Nicholas Ghasemi, Pouyan Gulati, Abhishek Harman, Mark Haluza, Zachary Honarkhah, Mehrdad Robert, Herve Liu, Jiacheng Liu, Weini Thummala, Sreeja Yang, Xiaoning Xin, Rui Zeng, Sophie |
| contents | We report on Just-in-Time catching test generation at Meta, designed to prevent bugs in large scale backend systems of hundreds of millions of line of code. Unlike traditional hardening tests, which pass at generation time, catching tests are meant to fail, surfacing bugs before code lands. The primary challenge is to reduce development drag from false positive test failures. Analyzing 22,126 generated tests, we show code-change-aware methods improve candidate catch generation 4x over hardening tests and 20x over coincidentally failing tests. To address false positives, we use rule-based and LLM-based assessors. These assessors reduce human review load by 70%. Inferential statistical analysis showed that human-accepted code changes are assessed to have significantly more false positives, while human-rejected changes have significantly more true positives. We reported 41 candidate catches to engineers; 8 were confirmed to be true positives, 4 of which would have led to serious failures had they remained uncaught. Overall, our results show that Just-in-Time catching is scalable, industrially applicable, and that it prevents serious failures from reaching production. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_22832 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Just-in-Time Catching Test Generation at Meta Becker, Matthew Chen, Yifei Cochran, Nicholas Ghasemi, Pouyan Gulati, Abhishek Harman, Mark Haluza, Zachary Honarkhah, Mehrdad Robert, Herve Liu, Jiacheng Liu, Weini Thummala, Sreeja Yang, Xiaoning Xin, Rui Zeng, Sophie Software Engineering Artificial Intelligence We report on Just-in-Time catching test generation at Meta, designed to prevent bugs in large scale backend systems of hundreds of millions of line of code. Unlike traditional hardening tests, which pass at generation time, catching tests are meant to fail, surfacing bugs before code lands. The primary challenge is to reduce development drag from false positive test failures. Analyzing 22,126 generated tests, we show code-change-aware methods improve candidate catch generation 4x over hardening tests and 20x over coincidentally failing tests. To address false positives, we use rule-based and LLM-based assessors. These assessors reduce human review load by 70%. Inferential statistical analysis showed that human-accepted code changes are assessed to have significantly more false positives, while human-rejected changes have significantly more true positives. We reported 41 candidate catches to engineers; 8 were confirmed to be true positives, 4 of which would have led to serious failures had they remained uncaught. Overall, our results show that Just-in-Time catching is scalable, industrially applicable, and that it prevents serious failures from reaching production. |
| title | Just-in-Time Catching Test Generation at Meta |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2601.22832 |