Alignment with Preference Optimization Is All You Need for LLM Safety
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929498523435008 |
|---|---|
| author | Alami, Reda Almansoori, Ali Khalifa Alzubaidi, Ahmed Seddik, Mohamed El Amine Farooq, Mugariya Hacid, Hakim |
| author_facet | Alami, Reda Almansoori, Ali Khalifa Alzubaidi, Ahmed Seddik, Mohamed El Amine Farooq, Mugariya Hacid, Hakim |
| contents | We demonstrate that preference optimization methods can effectively enhance LLM safety. Applying various alignment techniques to the Falcon 11B model using safety datasets, we achieve a significant boost in global safety score (from $57.64\%$ to $99.90\%$) as measured by LlamaGuard 3 8B, competing with state-of-the-art models. On toxicity benchmarks, average scores in adversarial settings dropped from over $0.6$ to less than $0.07$. However, this safety improvement comes at the cost of reduced general capabilities, particularly in math, suggesting a trade-off. We identify noise contrastive alignment (Safe-NCA) as an optimal method for balancing safety and performance. Our study ultimately shows that alignment techniques can be sufficient for building safe and robust models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_07772 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Alignment with Preference Optimization Is All You Need for LLM Safety Alami, Reda Almansoori, Ali Khalifa Alzubaidi, Ahmed Seddik, Mohamed El Amine Farooq, Mugariya Hacid, Hakim Machine Learning We demonstrate that preference optimization methods can effectively enhance LLM safety. Applying various alignment techniques to the Falcon 11B model using safety datasets, we achieve a significant boost in global safety score (from $57.64\%$ to $99.90\%$) as measured by LlamaGuard 3 8B, competing with state-of-the-art models. On toxicity benchmarks, average scores in adversarial settings dropped from over $0.6$ to less than $0.07$. However, this safety improvement comes at the cost of reduced general capabilities, particularly in math, suggesting a trade-off. We identify noise contrastive alignment (Safe-NCA) as an optimal method for balancing safety and performance. Our study ultimately shows that alignment techniques can be sufficient for building safe and robust models. |
| title | Alignment with Preference Optimization Is All You Need for LLM Safety |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2409.07772 |