iPanda: An LLM-based Agent for Automated Conformance Testing of Communication Protocols
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909709826523136 |
|---|---|
| author | Sun, Xikai Dang, Fan Jiang, Shiqi Xu, Jingao Liu, Kebin Miao, Xin Yang, Zihao Zhang, Weichen Lu, Haimo Zheng, Yawen Liu, Yunhao |
| author_facet | Sun, Xikai Dang, Fan Jiang, Shiqi Xu, Jingao Liu, Kebin Miao, Xin Yang, Zihao Zhang, Weichen Lu, Haimo Zheng, Yawen Liu, Yunhao |
| contents | Conformance testing is essential for ensuring that protocol implementations comply with their specifications. However, traditional testing approaches involve manually creating numerous test cases and scripts, making the process labor-intensive and inefficient. Recently, Large Language Models (LLMs) have demonstrated impressive text comprehension and code generation abilities, providing promising opportunities for automation. In this paper, we propose iPanda, the first framework that leverages LLMs to automate protocol conformance testing. Given a protocol specification document and its implementation, iPanda first employs a keyword-based method to automatically generate comprehensive test cases. Then, it utilizes retrieval-augmented generation and customized CoT strategy to effectively interpret the implementation and produce executable test programs. To further enhance programs' quality, iPanda incorporates an iterative optimization mechanism to refine generated test scripts interactively. Finally, by executing and analyzing the generated tests, iPanda systematically verifies compliance between implementations and protocol specifications. Comprehensive experiments on various protocols show that iPanda significantly outperforms pure LLM-based approaches, improving the success rate (Pass@1) of test-program generation by factors ranging from 4.675 times to 10.751 times. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_00378 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | iPanda: An LLM-based Agent for Automated Conformance Testing of Communication Protocols Sun, Xikai Dang, Fan Jiang, Shiqi Xu, Jingao Liu, Kebin Miao, Xin Yang, Zihao Zhang, Weichen Lu, Haimo Zheng, Yawen Liu, Yunhao Software Engineering Artificial Intelligence Conformance testing is essential for ensuring that protocol implementations comply with their specifications. However, traditional testing approaches involve manually creating numerous test cases and scripts, making the process labor-intensive and inefficient. Recently, Large Language Models (LLMs) have demonstrated impressive text comprehension and code generation abilities, providing promising opportunities for automation. In this paper, we propose iPanda, the first framework that leverages LLMs to automate protocol conformance testing. Given a protocol specification document and its implementation, iPanda first employs a keyword-based method to automatically generate comprehensive test cases. Then, it utilizes retrieval-augmented generation and customized CoT strategy to effectively interpret the implementation and produce executable test programs. To further enhance programs' quality, iPanda incorporates an iterative optimization mechanism to refine generated test scripts interactively. Finally, by executing and analyzing the generated tests, iPanda systematically verifies compliance between implementations and protocol specifications. Comprehensive experiments on various protocols show that iPanda significantly outperforms pure LLM-based approaches, improving the success rate (Pass@1) of test-program generation by factors ranging from 4.675 times to 10.751 times. |
| title | iPanda: An LLM-based Agent for Automated Conformance Testing of Communication Protocols |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2507.00378 |