Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917866346905600 |
|---|---|
| author | Samvelyan, Mikayel Raparthy, Sharath Chandra Lupu, Andrei Hambro, Eric Markosyan, Aram H. Bhatt, Manish Mao, Yuning Jiang, Minqi Parker-Holder, Jack Foerster, Jakob Rocktäschel, Tim Raileanu, Roberta |
| author_facet | Samvelyan, Mikayel Raparthy, Sharath Chandra Lupu, Andrei Hambro, Eric Markosyan, Aram H. Bhatt, Manish Mao, Yuning Jiang, Minqi Parker-Holder, Jack Foerster, Jakob Rocktäschel, Tim Raileanu, Roberta |
| contents | As large language models (LLMs) become increasingly prevalent across many real-world applications, understanding and enhancing their robustness to adversarial attacks is of paramount importance. Existing methods for identifying adversarial prompts tend to focus on specific domains, lack diversity, or require extensive human annotations. To address these limitations, we present Rainbow Teaming, a novel black-box approach for producing a diverse collection of adversarial prompts. Rainbow Teaming casts adversarial prompt generation as a quality-diversity problem and uses open-ended search to generate prompts that are both effective and diverse. Focusing on the safety domain, we use Rainbow Teaming to target various state-of-the-art LLMs, including the Llama 2 and Llama 3 models. Our approach reveals hundreds of effective adversarial prompts, with an attack success rate exceeding 90% across all tested models. Furthermore, we demonstrate that prompts generated by Rainbow Teaming are highly transferable and that fine-tuning models with synthetic data generated by our method significantly enhances their safety without sacrificing general performance or helpfulness. We additionally explore the versatility of Rainbow Teaming by applying it to question answering and cybersecurity, showcasing its potential to drive robust open-ended self-improvement in a wide range of applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_16822 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts Samvelyan, Mikayel Raparthy, Sharath Chandra Lupu, Andrei Hambro, Eric Markosyan, Aram H. Bhatt, Manish Mao, Yuning Jiang, Minqi Parker-Holder, Jack Foerster, Jakob Rocktäschel, Tim Raileanu, Roberta Computation and Language Artificial Intelligence Machine Learning As large language models (LLMs) become increasingly prevalent across many real-world applications, understanding and enhancing their robustness to adversarial attacks is of paramount importance. Existing methods for identifying adversarial prompts tend to focus on specific domains, lack diversity, or require extensive human annotations. To address these limitations, we present Rainbow Teaming, a novel black-box approach for producing a diverse collection of adversarial prompts. Rainbow Teaming casts adversarial prompt generation as a quality-diversity problem and uses open-ended search to generate prompts that are both effective and diverse. Focusing on the safety domain, we use Rainbow Teaming to target various state-of-the-art LLMs, including the Llama 2 and Llama 3 models. Our approach reveals hundreds of effective adversarial prompts, with an attack success rate exceeding 90% across all tested models. Furthermore, we demonstrate that prompts generated by Rainbow Teaming are highly transferable and that fine-tuning models with synthetic data generated by our method significantly enhances their safety without sacrificing general performance or helpfulness. We additionally explore the versatility of Rainbow Teaming by applying it to question answering and cybersecurity, showcasing its potential to drive robust open-ended self-improvement in a wide range of applications. |
| title | Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2402.16822 |