Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908375764172800 |
|---|---|
| author | Yuan, Youliang Jiao, Wenxiang Wang, Wenxuan Huang, Jen-tse Xu, Jiahao Liang, Tian He, Pinjia Tu, Zhaopeng |
| author_facet | Yuan, Youliang Jiao, Wenxiang Wang, Wenxuan Huang, Jen-tse Xu, Jiahao Liang, Tian He, Pinjia Tu, Zhaopeng |
| contents | This study addresses a critical gap in safety tuning practices for Large Language Models (LLMs) by identifying and tackling a refusal position bias within safety tuning data, which compromises the models' ability to appropriately refuse generating unsafe content. We introduce a novel approach, Decoupled Refusal Training (DeRTa), designed to empower LLMs to refuse compliance to harmful prompts at any response position, significantly enhancing their safety capabilities. DeRTa incorporates two novel components: (1) Maximum Likelihood Estimation (MLE) with Harmful Response Prefix, which trains models to recognize and avoid unsafe content by appending a segment of harmful response to the beginning of a safe response, and (2) Reinforced Transition Optimization (RTO), which equips models with the ability to transition from potential harm to safety refusal consistently throughout the harmful response sequence. Our empirical evaluation, conducted using LLaMA3 and Mistral model families across six attack scenarios, demonstrates that our method not only improves model safety without compromising performance but also surpasses baseline methods in defending against attacks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_09121 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training Yuan, Youliang Jiao, Wenxiang Wang, Wenxuan Huang, Jen-tse Xu, Jiahao Liang, Tian He, Pinjia Tu, Zhaopeng Computation and Language Artificial Intelligence This study addresses a critical gap in safety tuning practices for Large Language Models (LLMs) by identifying and tackling a refusal position bias within safety tuning data, which compromises the models' ability to appropriately refuse generating unsafe content. We introduce a novel approach, Decoupled Refusal Training (DeRTa), designed to empower LLMs to refuse compliance to harmful prompts at any response position, significantly enhancing their safety capabilities. DeRTa incorporates two novel components: (1) Maximum Likelihood Estimation (MLE) with Harmful Response Prefix, which trains models to recognize and avoid unsafe content by appending a segment of harmful response to the beginning of a safe response, and (2) Reinforced Transition Optimization (RTO), which equips models with the ability to transition from potential harm to safety refusal consistently throughout the harmful response sequence. Our empirical evaluation, conducted using LLaMA3 and Mistral model families across six attack scenarios, demonstrates that our method not only improves model safety without compromising performance but also surpasses baseline methods in defending against attacks. |
| title | Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2407.09121 |