Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909110254960640 |
|---|---|
| author | Chen, Kai Wang, Chunwei Yang, Kuo Han, Jianhua Hong, Lanqing Mi, Fei Xu, Hang Liu, Zhengying Huang, Wenyong Li, Zhenguo Yeung, Dit-Yan Shang, Lifeng Jiang, Xin Liu, Qun |
| author_facet | Chen, Kai Wang, Chunwei Yang, Kuo Han, Jianhua Hong, Lanqing Mi, Fei Xu, Hang Liu, Zhengying Huang, Wenyong Li, Zhenguo Yeung, Dit-Yan Shang, Lifeng Jiang, Xin Liu, Qun |
| contents | The rapid development of large language models (LLMs) has not only provided numerous opportunities but also presented significant challenges. This becomes particularly evident when LLMs inadvertently generate harmful or toxic content, either unintentionally or because of intentional inducement. Existing alignment methods usually direct LLMs toward the favorable outcomes by utilizing human-annotated, flawless instruction-response pairs. Conversely, this study proposes a novel alignment technique based on mistake analysis, which deliberately exposes LLMs to erroneous content to learn the reasons for mistakes and how to avoid them. In this case, mistakes are repurposed into valuable data for alignment, effectively helping to avoid the production of erroneous responses. Without external models or human annotations, our method leverages a model's intrinsic ability to discern undesirable mistakes and improves the safety of its generated responses. Experimental results reveal that our method outperforms existing alignment approaches in enhancing model safety while maintaining the overall utility. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_10477 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis Chen, Kai Wang, Chunwei Yang, Kuo Han, Jianhua Hong, Lanqing Mi, Fei Xu, Hang Liu, Zhengying Huang, Wenyong Li, Zhenguo Yeung, Dit-Yan Shang, Lifeng Jiang, Xin Liu, Qun Computation and Language Artificial Intelligence Machine Learning The rapid development of large language models (LLMs) has not only provided numerous opportunities but also presented significant challenges. This becomes particularly evident when LLMs inadvertently generate harmful or toxic content, either unintentionally or because of intentional inducement. Existing alignment methods usually direct LLMs toward the favorable outcomes by utilizing human-annotated, flawless instruction-response pairs. Conversely, this study proposes a novel alignment technique based on mistake analysis, which deliberately exposes LLMs to erroneous content to learn the reasons for mistakes and how to avoid them. In this case, mistakes are repurposed into valuable data for alignment, effectively helping to avoid the production of erroneous responses. Without external models or human annotations, our method leverages a model's intrinsic ability to discern undesirable mistakes and improves the safety of its generated responses. Experimental results reveal that our method outperforms existing alignment approaches in enhancing model safety while maintaining the overall utility. |
| title | Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2310.10477 |