Robot Collapse: Supply Chain Backdoor Attacks Against VLM-based Robotic Manipulation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908932121821184 |
|---|---|
| author | Wang, Xianlong Pan, Hewen Zhang, Hangtao Li, Minghui Hu, Shengshan Zhou, Ziqi Xue, Lulu Guo, Peijin Liu, Aishan Zhang, Leo Yu Jia, Xiaohua |
| author_facet | Wang, Xianlong Pan, Hewen Zhang, Hangtao Li, Minghui Hu, Shengshan Zhou, Ziqi Xue, Lulu Guo, Peijin Liu, Aishan Zhang, Leo Yu Jia, Xiaohua |
| contents | Robotic manipulation policies are increasingly empowered by \textit{large language models} (LLMs) and \textit{vision-language models} (VLMs), leveraging their understanding and perception capabilities. Recently, inference-time attacks against robotic manipulation have been extensively studied, yet backdoor attacks targeting model supply chain security in robotic policies remain largely unexplored. To fill this gap, we propose \texttt{TrojanRobot}, a backdoor injection framework for model supply chain attack scenarios, which embeds a malicious module into modular robotic policies via backdoor relationships to manipulate the LLM-to-VLM pathway and compromise the system. Our vanilla design instantiates this module as a backdoor-finetuned VLM. To further enhance attack performance, we propose a prime scheme by introducing the concept of \textit{LVLM-as-a-backdoor}, which leverages \textit{in-context instruction learning} (ICIL) to steer \textit{large vision-language model} (LVLM) behavior through backdoored system prompts. Moreover, we develop three types of prime attacks, \textit{permutation}, \textit{stagnation}, and \textit{intentional}, achieving flexible backdoor attack effects. Extensive physical-world and simulator experiments on 18 real-world manipulation tasks and 4 VLMs verify the superiority of proposed \texttt{TrojanRobot} |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_11683 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Robot Collapse: Supply Chain Backdoor Attacks Against VLM-based Robotic Manipulation Wang, Xianlong Pan, Hewen Zhang, Hangtao Li, Minghui Hu, Shengshan Zhou, Ziqi Xue, Lulu Guo, Peijin Liu, Aishan Zhang, Leo Yu Jia, Xiaohua Robotics Artificial Intelligence Robotic manipulation policies are increasingly empowered by \textit{large language models} (LLMs) and \textit{vision-language models} (VLMs), leveraging their understanding and perception capabilities. Recently, inference-time attacks against robotic manipulation have been extensively studied, yet backdoor attacks targeting model supply chain security in robotic policies remain largely unexplored. To fill this gap, we propose \texttt{TrojanRobot}, a backdoor injection framework for model supply chain attack scenarios, which embeds a malicious module into modular robotic policies via backdoor relationships to manipulate the LLM-to-VLM pathway and compromise the system. Our vanilla design instantiates this module as a backdoor-finetuned VLM. To further enhance attack performance, we propose a prime scheme by introducing the concept of \textit{LVLM-as-a-backdoor}, which leverages \textit{in-context instruction learning} (ICIL) to steer \textit{large vision-language model} (LVLM) behavior through backdoored system prompts. Moreover, we develop three types of prime attacks, \textit{permutation}, \textit{stagnation}, and \textit{intentional}, achieving flexible backdoor attack effects. Extensive physical-world and simulator experiments on 18 real-world manipulation tasks and 4 VLMs verify the superiority of proposed \texttt{TrojanRobot} |
| title | Robot Collapse: Supply Chain Backdoor Attacks Against VLM-based Robotic Manipulation |
| topic | Robotics Artificial Intelligence |
| url | https://arxiv.org/abs/2411.11683 |