Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915335608729600 |
|---|---|
| author | An, Bang Zhu, Sicheng Zhang, Ruiyi Panaitescu-Liess, Michael-Andrei Xu, Yuancheng Huang, Furong |
| author_facet | An, Bang Zhu, Sicheng Zhang, Ruiyi Panaitescu-Liess, Michael-Andrei Xu, Yuancheng Huang, Furong |
| contents | Safety-aligned large language models (LLMs) sometimes falsely refuse pseudo-harmful prompts, like "how to kill a mosquito," which are actually harmless. Frequent false refusals not only frustrate users but also provoke a public backlash against the very values alignment seeks to protect. In this paper, we propose the first method to auto-generate diverse, content-controlled, and model-dependent pseudo-harmful prompts. Using this method, we construct an evaluation dataset called PHTest, which is ten times larger than existing datasets, covers more false refusal patterns, and separately labels controversial prompts. We evaluate 20 LLMs on PHTest, uncovering new insights due to its scale and labeling. Our findings reveal a trade-off between minimizing false refusals and improving safety against jailbreak attacks. Moreover, we show that many jailbreak defenses significantly increase the false refusal rates, thereby undermining usability. Our method and dataset can help developers evaluate and fine-tune safer and more usable LLMs. Our code and dataset are available at https://github.com/umd-huang-lab/FalseRefusal |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_00598 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Models An, Bang Zhu, Sicheng Zhang, Ruiyi Panaitescu-Liess, Michael-Andrei Xu, Yuancheng Huang, Furong Computation and Language Cryptography and Security Computers and Society Machine Learning Safety-aligned large language models (LLMs) sometimes falsely refuse pseudo-harmful prompts, like "how to kill a mosquito," which are actually harmless. Frequent false refusals not only frustrate users but also provoke a public backlash against the very values alignment seeks to protect. In this paper, we propose the first method to auto-generate diverse, content-controlled, and model-dependent pseudo-harmful prompts. Using this method, we construct an evaluation dataset called PHTest, which is ten times larger than existing datasets, covers more false refusal patterns, and separately labels controversial prompts. We evaluate 20 LLMs on PHTest, uncovering new insights due to its scale and labeling. Our findings reveal a trade-off between minimizing false refusals and improving safety against jailbreak attacks. Moreover, we show that many jailbreak defenses significantly increase the false refusal rates, thereby undermining usability. Our method and dataset can help developers evaluate and fine-tune safer and more usable LLMs. Our code and dataset are available at https://github.com/umd-huang-lab/FalseRefusal |
| title | Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Models |
| topic | Computation and Language Cryptography and Security Computers and Society Machine Learning |
| url | https://arxiv.org/abs/2409.00598 |