Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2406.09187 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913864589770752 |
|---|---|
| author | Xiang, Zhen Zheng, Linzhi Li, Yanjie Hong, Junyuan Li, Qinbin Xie, Han Zhang, Jiawei Xiong, Zidi Xie, Chulin Yang, Carl Song, Dawn Li, Bo |
| author_facet | Xiang, Zhen Zheng, Linzhi Li, Yanjie Hong, Junyuan Li, Qinbin Xie, Han Zhang, Jiawei Xiong, Zidi Xie, Chulin Yang, Carl Song, Dawn Li, Bo |
| contents | The rapid advancement of large language model (LLM) agents has raised new concerns regarding their safety and security. In this paper, we propose GuardAgent, the first guardrail agent to protect target agents by dynamically checking whether their actions satisfy given safety guard requests. Specifically, GuardAgent first analyzes the safety guard requests to generate a task plan, and then maps this plan into guardrail code for execution. By performing the code execution, GuardAgent can deterministically follow the safety guard request and safeguard target agents. In both steps, an LLM is utilized as the reasoning component, supplemented by in-context demonstrations retrieved from a memory module storing experiences from previous tasks. In addition, we propose two novel benchmarks: EICU-AC benchmark to assess the access control for healthcare agents and Mind2Web-SC benchmark to evaluate the safety policies for web agents. We show that GuardAgent effectively moderates the violation actions for different types of agents on these two benchmarks with over 98% and 83% guardrail accuracies, respectively. Project page: https://guardagent.github.io/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_09187 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning Xiang, Zhen Zheng, Linzhi Li, Yanjie Hong, Junyuan Li, Qinbin Xie, Han Zhang, Jiawei Xiong, Zidi Xie, Chulin Yang, Carl Song, Dawn Li, Bo Machine Learning The rapid advancement of large language model (LLM) agents has raised new concerns regarding their safety and security. In this paper, we propose GuardAgent, the first guardrail agent to protect target agents by dynamically checking whether their actions satisfy given safety guard requests. Specifically, GuardAgent first analyzes the safety guard requests to generate a task plan, and then maps this plan into guardrail code for execution. By performing the code execution, GuardAgent can deterministically follow the safety guard request and safeguard target agents. In both steps, an LLM is utilized as the reasoning component, supplemented by in-context demonstrations retrieved from a memory module storing experiences from previous tasks. In addition, we propose two novel benchmarks: EICU-AC benchmark to assess the access control for healthcare agents and Mind2Web-SC benchmark to evaluate the safety policies for web agents. We show that GuardAgent effectively moderates the violation actions for different types of agents on these two benchmarks with over 98% and 83% guardrail accuracies, respectively. Project page: https://guardagent.github.io/ |
| title | GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2406.09187 |