LeCov: Multi-level Testing Criteria for Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916362770710528 |
|---|---|
| author | Xie, Xuan Song, Jiayang Huang, Yuheng Song, Da Zhang, Fuyuan Juefei-Xu, Felix Ma, Lei |
| author_facet | Xie, Xuan Song, Jiayang Huang, Yuheng Song, Da Zhang, Fuyuan Juefei-Xu, Felix Ma, Lei |
| contents | Large Language Models (LLMs) are widely used in many different domains, but because of their limited interpretability, there are questions about how trustworthy they are in various perspectives, e.g., truthfulness and toxicity. Recent research has started developing testing methods for LLMs, aiming to uncover untrustworthy issues, i.e., defects, before deployment. However, systematic and formalized testing criteria are lacking, which hinders a comprehensive assessment of the extent and adequacy of testing exploration. To mitigate this threat, we propose a set of multi-level testing criteria, LeCov, for LLMs. The criteria consider three crucial LLM internal components, i.e., the attention mechanism, feed-forward neurons, and uncertainty, and contain nine types of testing criteria in total. We apply the criteria in two scenarios: test prioritization and coverage-guided testing. The experiment evaluation, on three models and four datasets, demonstrates the usefulness and effectiveness of LeCov. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_10474 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LeCov: Multi-level Testing Criteria for Large Language Models Xie, Xuan Song, Jiayang Huang, Yuheng Song, Da Zhang, Fuyuan Juefei-Xu, Felix Ma, Lei Software Engineering Artificial Intelligence Computation and Language Cryptography and Security Machine Learning Large Language Models (LLMs) are widely used in many different domains, but because of their limited interpretability, there are questions about how trustworthy they are in various perspectives, e.g., truthfulness and toxicity. Recent research has started developing testing methods for LLMs, aiming to uncover untrustworthy issues, i.e., defects, before deployment. However, systematic and formalized testing criteria are lacking, which hinders a comprehensive assessment of the extent and adequacy of testing exploration. To mitigate this threat, we propose a set of multi-level testing criteria, LeCov, for LLMs. The criteria consider three crucial LLM internal components, i.e., the attention mechanism, feed-forward neurons, and uncertainty, and contain nine types of testing criteria in total. We apply the criteria in two scenarios: test prioritization and coverage-guided testing. The experiment evaluation, on three models and four datasets, demonstrates the usefulness and effectiveness of LeCov. |
| title | LeCov: Multi-level Testing Criteria for Large Language Models |
| topic | Software Engineering Artificial Intelligence Computation and Language Cryptography and Security Machine Learning |
| url | https://arxiv.org/abs/2408.10474 |