Self and Cross-Model Distillation for LLMs: Effective Methods for Refusal Pattern Alignment
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910723562536960 |
|---|---|
| author | Li, Jie Liu, Yi Liu, Chongyang Ren, Xiaoning Shi, Ling Sun, Weisong Xue, Yinxing |
| author_facet | Li, Jie Liu, Yi Liu, Chongyang Ren, Xiaoning Shi, Ling Sun, Weisong Xue, Yinxing |
| contents | Large Language Models (LLMs) like OpenAI's GPT series, Anthropic's Claude, and Meta's LLaMa have shown remarkable capabilities in text generation. However, their susceptibility to toxic prompts presents significant security challenges. This paper investigates alignment techniques, including Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), to mitigate these risks. We conduct an empirical study on refusal patterns across nine LLMs, revealing that models with uniform refusal patterns, such as Claude3, exhibit higher security. Based on these findings, we propose self-distilling and cross-model distilling methods to enhance LLM security. Our results show that these methods significantly improve refusal rates and reduce unsafe content, with cross-model distilling achieving refusal rates close to Claude3's 94.51%. These findings underscore the potential of distillation-based alignment in securing LLMs against toxic prompts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_11285 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Self and Cross-Model Distillation for LLMs: Effective Methods for Refusal Pattern Alignment Li, Jie Liu, Yi Liu, Chongyang Ren, Xiaoning Shi, Ling Sun, Weisong Xue, Yinxing Cryptography and Security Computation and Language Large Language Models (LLMs) like OpenAI's GPT series, Anthropic's Claude, and Meta's LLaMa have shown remarkable capabilities in text generation. However, their susceptibility to toxic prompts presents significant security challenges. This paper investigates alignment techniques, including Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), to mitigate these risks. We conduct an empirical study on refusal patterns across nine LLMs, revealing that models with uniform refusal patterns, such as Claude3, exhibit higher security. Based on these findings, we propose self-distilling and cross-model distilling methods to enhance LLM security. Our results show that these methods significantly improve refusal rates and reduce unsafe content, with cross-model distilling achieving refusal rates close to Claude3's 94.51%. These findings underscore the potential of distillation-based alignment in securing LLMs against toxic prompts. |
| title | Self and Cross-Model Distillation for LLMs: Effective Methods for Refusal Pattern Alignment |
| topic | Cryptography and Security Computation and Language |
| url | https://arxiv.org/abs/2406.11285 |