Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2501.07238 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912185998901248 |
|---|---|
| author | Bullwinkel, Blake Minnich, Amanda Chawla, Shiven Lopez, Gary Pouliot, Martin Maxwell, Whitney de Gruyter, Joris Pratt, Katherine Qi, Saphir Chikanov, Nina Lutz, Roman Dheekonda, Raja Sekhar Rao Jagdagdorj, Bolor-Erdene Kim, Eugenia Song, Justin Hines, Keegan Jones, Daniel Severi, Giorgio Lundeen, Richard Vaughan, Sam Westerhoff, Victoria Bryan, Pete Kumar, Ram Shankar Siva Zunger, Yonatan Kawaguchi, Chang Russinovich, Mark |
| author_facet | Bullwinkel, Blake Minnich, Amanda Chawla, Shiven Lopez, Gary Pouliot, Martin Maxwell, Whitney de Gruyter, Joris Pratt, Katherine Qi, Saphir Chikanov, Nina Lutz, Roman Dheekonda, Raja Sekhar Rao Jagdagdorj, Bolor-Erdene Kim, Eugenia Song, Justin Hines, Keegan Jones, Daniel Severi, Giorgio Lundeen, Richard Vaughan, Sam Westerhoff, Victoria Bryan, Pete Kumar, Ram Shankar Siva Zunger, Yonatan Kawaguchi, Chang Russinovich, Mark |
| contents | In recent years, AI red teaming has emerged as a practice for probing the safety and security of generative AI systems. Due to the nascency of the field, there are many open questions about how red teaming operations should be conducted. Based on our experience red teaming over 100 generative AI products at Microsoft, we present our internal threat model ontology and eight main lessons we have learned:
1. Understand what the system can do and where it is applied
2. You don't have to compute gradients to break an AI system
3. AI red teaming is not safety benchmarking
4. Automation can help cover more of the risk landscape
5. The human element of AI red teaming is crucial
6. Responsible AI harms are pervasive but difficult to measure
7. LLMs amplify existing security risks and introduce new ones
8. The work of securing AI systems will never be complete
By sharing these insights alongside case studies from our operations, we offer practical recommendations aimed at aligning red teaming efforts with real world risks. We also highlight aspects of AI red teaming that we believe are often misunderstood and discuss open questions for the field to consider. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_07238 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Lessons From Red Teaming 100 Generative AI Products Bullwinkel, Blake Minnich, Amanda Chawla, Shiven Lopez, Gary Pouliot, Martin Maxwell, Whitney de Gruyter, Joris Pratt, Katherine Qi, Saphir Chikanov, Nina Lutz, Roman Dheekonda, Raja Sekhar Rao Jagdagdorj, Bolor-Erdene Kim, Eugenia Song, Justin Hines, Keegan Jones, Daniel Severi, Giorgio Lundeen, Richard Vaughan, Sam Westerhoff, Victoria Bryan, Pete Kumar, Ram Shankar Siva Zunger, Yonatan Kawaguchi, Chang Russinovich, Mark Artificial Intelligence In recent years, AI red teaming has emerged as a practice for probing the safety and security of generative AI systems. Due to the nascency of the field, there are many open questions about how red teaming operations should be conducted. Based on our experience red teaming over 100 generative AI products at Microsoft, we present our internal threat model ontology and eight main lessons we have learned: 1. Understand what the system can do and where it is applied 2. You don't have to compute gradients to break an AI system 3. AI red teaming is not safety benchmarking 4. Automation can help cover more of the risk landscape 5. The human element of AI red teaming is crucial 6. Responsible AI harms are pervasive but difficult to measure 7. LLMs amplify existing security risks and introduce new ones 8. The work of securing AI systems will never be complete By sharing these insights alongside case studies from our operations, we offer practical recommendations aimed at aligning red teaming efforts with real world risks. We also highlight aspects of AI red teaming that we believe are often misunderstood and discuss open questions for the field to consider. |
| title | Lessons From Red Teaming 100 Generative AI Products |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2501.07238 |