Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.18676 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912225113931776 |
|---|---|
| author | Karnik, Sathwik Hong, Zhang-Wei Abhangi, Nishant Lin, Yen-Chen Wang, Tsun-Hsuan Dupuy, Christophe Gupta, Rahul Agrawal, Pulkit |
| author_facet | Karnik, Sathwik Hong, Zhang-Wei Abhangi, Nishant Lin, Yen-Chen Wang, Tsun-Hsuan Dupuy, Christophe Gupta, Rahul Agrawal, Pulkit |
| contents | Language-conditioned robot models have the potential to enable robots to perform a wide range of tasks based on natural language instructions. However, assessing their safety and effectiveness remains challenging because it is difficult to test all the different ways a single task can be phrased. Current benchmarks have two key limitations: they rely on a limited set of human-generated instructions, missing many challenging cases, and focus only on task performance without assessing safety, such as avoiding damage. To address these gaps, we introduce Embodied Red Teaming (ERT), a new evaluation method that generates diverse and challenging instructions to test these models. ERT uses automated red teaming techniques with Vision Language Models (VLMs) to create contextually grounded, difficult instructions. Experimental results show that state-of-the-art language-conditioned robot models fail or behave unsafely on ERT-generated instructions, underscoring the shortcomings of current benchmarks in evaluating real-world performance and safety. Code and videos are available at: https://s-karnik.github.io/embodied-red-team-project-page. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_18676 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Embodied Red Teaming for Auditing Robotic Foundation Models Karnik, Sathwik Hong, Zhang-Wei Abhangi, Nishant Lin, Yen-Chen Wang, Tsun-Hsuan Dupuy, Christophe Gupta, Rahul Agrawal, Pulkit Robotics Artificial Intelligence Machine Learning Language-conditioned robot models have the potential to enable robots to perform a wide range of tasks based on natural language instructions. However, assessing their safety and effectiveness remains challenging because it is difficult to test all the different ways a single task can be phrased. Current benchmarks have two key limitations: they rely on a limited set of human-generated instructions, missing many challenging cases, and focus only on task performance without assessing safety, such as avoiding damage. To address these gaps, we introduce Embodied Red Teaming (ERT), a new evaluation method that generates diverse and challenging instructions to test these models. ERT uses automated red teaming techniques with Vision Language Models (VLMs) to create contextually grounded, difficult instructions. Experimental results show that state-of-the-art language-conditioned robot models fail or behave unsafely on ERT-generated instructions, underscoring the shortcomings of current benchmarks in evaluating real-world performance and safety. Code and videos are available at: https://s-karnik.github.io/embodied-red-team-project-page. |
| title | Embodied Red Teaming for Auditing Robotic Foundation Models |
| topic | Robotics Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2411.18676 |