Are LLMs Correctly Integrated into Software Systems?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916603892858880 |
|---|---|
| author | Shao, Yuchen Huang, Yuheng Shen, Jiawei Ma, Lei Su, Ting Wan, Chengcheng |
| author_facet | Shao, Yuchen Huang, Yuheng Shen, Jiawei Ma, Lei Su, Ting Wan, Chengcheng |
| contents | Large language models (LLMs) provide effective solutions in various application scenarios, with the support of retrieval-augmented generation (RAG). However, developers face challenges in integrating LLM and RAG into software systems, due to lacking interface specifications, various requirements from software context, and complicated system management. In this paper, we have conducted a comprehensive study of 100 open-source applications that incorporate LLMs with RAG support, and identified 18 defect patterns. Our study reveals that 77% of these applications contain more than three types of integration defects that degrade software functionality, efficiency, and security. Guided by our study, we propose systematic guidelines for resolving these defects in software life cycle. We also construct an open-source defect library Hydrangea. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_05138 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Are LLMs Correctly Integrated into Software Systems? Shao, Yuchen Huang, Yuheng Shen, Jiawei Ma, Lei Su, Ting Wan, Chengcheng Software Engineering Artificial Intelligence Large language models (LLMs) provide effective solutions in various application scenarios, with the support of retrieval-augmented generation (RAG). However, developers face challenges in integrating LLM and RAG into software systems, due to lacking interface specifications, various requirements from software context, and complicated system management. In this paper, we have conducted a comprehensive study of 100 open-source applications that incorporate LLMs with RAG support, and identified 18 defect patterns. Our study reveals that 77% of these applications contain more than three types of integration defects that degrade software functionality, efficiency, and security. Guided by our study, we propose systematic guidelines for resolving these defects in software life cycle. We also construct an open-source defect library Hydrangea. |
| title | Are LLMs Correctly Integrated into Software Systems? |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2407.05138 |