On the Vulnerability of Applying Retrieval-Augmented Generation within Knowledge-Intensive Application Domains
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910974366187520 |
|---|---|
| author | Xian, Xun Wang, Ganghua Bi, Xuan Srinivasa, Jayanth Kundu, Ashish Fleming, Charles Hong, Mingyi Ding, Jie |
| author_facet | Xian, Xun Wang, Ganghua Bi, Xuan Srinivasa, Jayanth Kundu, Ashish Fleming, Charles Hong, Mingyi Ding, Jie |
| contents | Retrieval-Augmented Generation (RAG) has been empirically shown to enhance the performance of large language models (LLMs) in knowledge-intensive domains such as healthcare, finance, and legal contexts. Given a query, RAG retrieves relevant documents from a corpus and integrates them into the LLMs' generation process. In this study, we investigate the adversarial robustness of RAG, focusing specifically on examining the retrieval system. First, across 225 different setup combinations of corpus, retriever, query, and targeted information, we show that retrieval systems are vulnerable to universal poisoning attacks in medical Q\&A. In such attacks, adversaries generate poisoned documents containing a broad spectrum of targeted information, such as personally identifiable information. When these poisoned documents are inserted into a corpus, they can be accurately retrieved by any users, as long as attacker-specified queries are used. To understand this vulnerability, we discovered that the deviation from the query's embedding to that of the poisoned document tends to follow a pattern in which the high similarity between the poisoned document and the query is retained, thereby enabling precise retrieval. Based on these findings, we develop a new detection-based defense to ensure the safe use of RAG. Through extensive experiments spanning various Q\&A domains, we observed that our proposed method consistently achieves excellent detection rates in nearly all cases. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_17275 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On the Vulnerability of Applying Retrieval-Augmented Generation within Knowledge-Intensive Application Domains Xian, Xun Wang, Ganghua Bi, Xuan Srinivasa, Jayanth Kundu, Ashish Fleming, Charles Hong, Mingyi Ding, Jie Cryptography and Security Artificial Intelligence Computation and Language Databases Emerging Technologies Information Retrieval Machine Learning 68T50, 68T05, 94A60 I.2.7; K.6.5; H.3.3; I.2.6 Retrieval-Augmented Generation (RAG) has been empirically shown to enhance the performance of large language models (LLMs) in knowledge-intensive domains such as healthcare, finance, and legal contexts. Given a query, RAG retrieves relevant documents from a corpus and integrates them into the LLMs' generation process. In this study, we investigate the adversarial robustness of RAG, focusing specifically on examining the retrieval system. First, across 225 different setup combinations of corpus, retriever, query, and targeted information, we show that retrieval systems are vulnerable to universal poisoning attacks in medical Q\&A. In such attacks, adversaries generate poisoned documents containing a broad spectrum of targeted information, such as personally identifiable information. When these poisoned documents are inserted into a corpus, they can be accurately retrieved by any users, as long as attacker-specified queries are used. To understand this vulnerability, we discovered that the deviation from the query's embedding to that of the poisoned document tends to follow a pattern in which the high similarity between the poisoned document and the query is retained, thereby enabling precise retrieval. Based on these findings, we develop a new detection-based defense to ensure the safe use of RAG. Through extensive experiments spanning various Q\&A domains, we observed that our proposed method consistently achieves excellent detection rates in nearly all cases. |
| title | On the Vulnerability of Applying Retrieval-Augmented Generation within Knowledge-Intensive Application Domains |
| topic | Cryptography and Security Artificial Intelligence Computation and Language Databases Emerging Technologies Information Retrieval Machine Learning 68T50, 68T05, 94A60 I.2.7; K.6.5; H.3.3; I.2.6 |
| url | https://arxiv.org/abs/2409.17275 |