RM -RF: Reward Model for Run-Free Unit Test Evaluation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908774495682560 |
|---|---|
| author | Bruches, Elena Grebenkin, Daniil Klementev, Mikhail Alperovich, Vadim Derunets, Roman Baturova, Dari Mkrtchyan, Georgy Sedukhin, Oleg Bondarenko, Ivan Bushkov, Nikolay Moiseev, Stanislav |
| author_facet | Bruches, Elena Grebenkin, Daniil Klementev, Mikhail Alperovich, Vadim Derunets, Roman Baturova, Dari Mkrtchyan, Georgy Sedukhin, Oleg Bondarenko, Ivan Bushkov, Nikolay Moiseev, Stanislav |
| contents | We present RM-RF, a lightweight reward model for run-free evaluation of automatically generated unit tests. Instead of repeatedly compiling and executing candidate tests, RM-RF predicts - from source and test code alone - three execution-derived signals: (1) whether the augmented test suite compiles and runs successfully, (2) whether the generated test cases increase code coverage, and (3) whether the generated test cases improve the mutation kill rate. To train and evaluate RM-RF we assemble a multilingual dataset (Java, Python, Go) of focal files, test files, and candidate test additions labeled by an execution-based pipeline, and we release an associated dataset and methodology for comparative evaluation. We tested multiple model families and tuning regimes (zero-shot, full fine-tuning, and PEFT via LoRA), achieving an average F1 of 0.69 across the three targets. Compared to conventional compile-and-run instruments, RM-RF provides substantially lower latency and infrastructure cost while delivering competitive predictive fidelity, enabling fast, scalable feedback for large-scale test generation and RL-based code optimization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_13097 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | RM -RF: Reward Model for Run-Free Unit Test Evaluation Bruches, Elena Grebenkin, Daniil Klementev, Mikhail Alperovich, Vadim Derunets, Roman Baturova, Dari Mkrtchyan, Georgy Sedukhin, Oleg Bondarenko, Ivan Bushkov, Nikolay Moiseev, Stanislav Software Engineering Machine Learning We present RM-RF, a lightweight reward model for run-free evaluation of automatically generated unit tests. Instead of repeatedly compiling and executing candidate tests, RM-RF predicts - from source and test code alone - three execution-derived signals: (1) whether the augmented test suite compiles and runs successfully, (2) whether the generated test cases increase code coverage, and (3) whether the generated test cases improve the mutation kill rate. To train and evaluate RM-RF we assemble a multilingual dataset (Java, Python, Go) of focal files, test files, and candidate test additions labeled by an execution-based pipeline, and we release an associated dataset and methodology for comparative evaluation. We tested multiple model families and tuning regimes (zero-shot, full fine-tuning, and PEFT via LoRA), achieving an average F1 of 0.69 across the three targets. Compared to conventional compile-and-run instruments, RM-RF provides substantially lower latency and infrastructure cost while delivering competitive predictive fidelity, enabling fast, scalable feedback for large-scale test generation and RL-based code optimization. |
| title | RM -RF: Reward Model for Run-Free Unit Test Evaluation |
| topic | Software Engineering Machine Learning |
| url | https://arxiv.org/abs/2601.13097 |