Automatic Generation of Behavioral Test Cases For Natural Language Processing Using Clustering and Prompting
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913461337849856 |
|---|---|
| author | Li, Ying Singh, Rahul Joshi, Tarun Sudjianto, Agus |
| author_facet | Li, Ying Singh, Rahul Joshi, Tarun Sudjianto, Agus |
| contents | Recent work in behavioral testing for natural language processing (NLP) models, such as Checklist, is inspired by related paradigms in software engineering testing. They allow evaluation of general linguistic capabilities and domain understanding, hence can help evaluate conceptual soundness and identify model weaknesses. However, a major challenge is the creation of test cases. The current packages rely on semi-automated approach using manual development which requires domain expertise and can be time consuming. This paper introduces an automated approach to develop test cases by exploiting the power of large language models and statistical techniques. It clusters the text representations to carefully construct meaningful groups and then apply prompting techniques to automatically generate Minimal Functionality Tests (MFT). The well-known Amazon Reviews corpus is used to demonstrate our approach. We analyze the behavioral test profiles across four different classification algorithms and discuss the limitations and strengths of those models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_00161 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Automatic Generation of Behavioral Test Cases For Natural Language Processing Using Clustering and Prompting Li, Ying Singh, Rahul Joshi, Tarun Sudjianto, Agus Computation and Language Artificial Intelligence Emerging Technologies Machine Learning Recent work in behavioral testing for natural language processing (NLP) models, such as Checklist, is inspired by related paradigms in software engineering testing. They allow evaluation of general linguistic capabilities and domain understanding, hence can help evaluate conceptual soundness and identify model weaknesses. However, a major challenge is the creation of test cases. The current packages rely on semi-automated approach using manual development which requires domain expertise and can be time consuming. This paper introduces an automated approach to develop test cases by exploiting the power of large language models and statistical techniques. It clusters the text representations to carefully construct meaningful groups and then apply prompting techniques to automatically generate Minimal Functionality Tests (MFT). The well-known Amazon Reviews corpus is used to demonstrate our approach. We analyze the behavioral test profiles across four different classification algorithms and discuss the limitations and strengths of those models. |
| title | Automatic Generation of Behavioral Test Cases For Natural Language Processing Using Clustering and Prompting |
| topic | Computation and Language Artificial Intelligence Emerging Technologies Machine Learning |
| url | https://arxiv.org/abs/2408.00161 |