Optimizing Case-Based Reasoning System for Functional Test Script Generation with Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916761327108096 |
|---|---|
| author | Guo, Siyuan Liu, Huiwu Chen, Xiaolong Xie, Yuming Zhang, Liang Han, Tao Chen, Hechang Chang, Yi Wang, Jun |
| author_facet | Guo, Siyuan Liu, Huiwu Chen, Xiaolong Xie, Yuming Zhang, Liang Han, Tao Chen, Hechang Chang, Yi Wang, Jun |
| contents | In this work, we explore the potential of large language models (LLMs) for generating functional test scripts, which necessitates understanding the dynamically evolving code structure of the target software. To achieve this, we propose a case-based reasoning (CBR) system utilizing a 4R cycle (i.e., retrieve, reuse, revise, and retain), which maintains and leverages a case bank of test intent descriptions and corresponding test scripts to facilitate LLMs for test script generation. To improve user experience further, we introduce Re4, an optimization method for the CBR system, comprising reranking-based retrieval finetuning and reinforced reuse finetuning. Specifically, we first identify positive examples with high semantic and script similarity, providing reliable pseudo-labels for finetuning the retriever model without costly labeling. Then, we apply supervised finetuning, followed by a reinforcement learning finetuning stage, to align LLMs with our production scenarios, ensuring the faithful reuse of retrieved cases. Extensive experimental results on two product development units from Huawei Datacom demonstrate the superiority of the proposed CBR+Re4. Notably, we also show that the proposed Re4 method can help alleviate the repetitive generation issues with LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_20576 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Optimizing Case-Based Reasoning System for Functional Test Script Generation with Large Language Models Guo, Siyuan Liu, Huiwu Chen, Xiaolong Xie, Yuming Zhang, Liang Han, Tao Chen, Hechang Chang, Yi Wang, Jun Software Engineering Computation and Language Machine Learning In this work, we explore the potential of large language models (LLMs) for generating functional test scripts, which necessitates understanding the dynamically evolving code structure of the target software. To achieve this, we propose a case-based reasoning (CBR) system utilizing a 4R cycle (i.e., retrieve, reuse, revise, and retain), which maintains and leverages a case bank of test intent descriptions and corresponding test scripts to facilitate LLMs for test script generation. To improve user experience further, we introduce Re4, an optimization method for the CBR system, comprising reranking-based retrieval finetuning and reinforced reuse finetuning. Specifically, we first identify positive examples with high semantic and script similarity, providing reliable pseudo-labels for finetuning the retriever model without costly labeling. Then, we apply supervised finetuning, followed by a reinforcement learning finetuning stage, to align LLMs with our production scenarios, ensuring the faithful reuse of retrieved cases. Extensive experimental results on two product development units from Huawei Datacom demonstrate the superiority of the proposed CBR+Re4. Notably, we also show that the proposed Re4 method can help alleviate the repetitive generation issues with LLMs. |
| title | Optimizing Case-Based Reasoning System for Functional Test Script Generation with Large Language Models |
| topic | Software Engineering Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2503.20576 |