To Protect the LLM Agent Against the Prompt Injection Attack with Polymorphic Prompt
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916781878149120 |
|---|---|
| author | Wang, Zhilong Nagaraja, Neha Zhang, Lan Bahsi, Hayretdin Patil, Pawan Liu, Peng |
| author_facet | Wang, Zhilong Nagaraja, Neha Zhang, Lan Bahsi, Hayretdin Patil, Pawan Liu, Peng |
| contents | LLM agents are widely used as agents for customer support, content generation, and code assistance. However, they are vulnerable to prompt injection attacks, where adversarial inputs manipulate the model's behavior. Traditional defenses like input sanitization, guard models, and guardrails are either cumbersome or ineffective. In this paper, we propose a novel, lightweight defense mechanism called Polymorphic Prompt Assembling (PPA), which protects against prompt injection with near-zero overhead. The approach is based on the insight that prompt injection requires guessing and breaking the structure of the system prompt. By dynamically varying the structure of system prompts, PPA prevents attackers from predicting the prompt structure, thereby enhancing security without compromising performance. We conducted experiments to evaluate the effectiveness of PPA against existing attacks and compared it with other defense methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_05739 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | To Protect the LLM Agent Against the Prompt Injection Attack with Polymorphic Prompt Wang, Zhilong Nagaraja, Neha Zhang, Lan Bahsi, Hayretdin Patil, Pawan Liu, Peng Cryptography and Security Artificial Intelligence LLM agents are widely used as agents for customer support, content generation, and code assistance. However, they are vulnerable to prompt injection attacks, where adversarial inputs manipulate the model's behavior. Traditional defenses like input sanitization, guard models, and guardrails are either cumbersome or ineffective. In this paper, we propose a novel, lightweight defense mechanism called Polymorphic Prompt Assembling (PPA), which protects against prompt injection with near-zero overhead. The approach is based on the insight that prompt injection requires guessing and breaking the structure of the system prompt. By dynamically varying the structure of system prompts, PPA prevents attackers from predicting the prompt structure, thereby enhancing security without compromising performance. We conducted experiments to evaluate the effectiveness of PPA against existing attacks and compared it with other defense methods. |
| title | To Protect the LLM Agent Against the Prompt Injection Attack with Polymorphic Prompt |
| topic | Cryptography and Security Artificial Intelligence |
| url | https://arxiv.org/abs/2506.05739 |