Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.18341 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916179934707712 |
|---|---|
| author | Chen, Xiusi Wen, Hongzhi Nag, Sreyashi Luo, Chen Yin, Qingyu Li, Ruirui Li, Zheng Wang, Wei |
| author_facet | Chen, Xiusi Wen, Hongzhi Nag, Sreyashi Luo, Chen Yin, Qingyu Li, Ruirui Li, Zheng Wang, Wei |
| contents | With the rapid development of large language models (LLMs), aligning LLMs with human values and societal norms to ensure their reliability and safety has become crucial. Reinforcement learning with human feedback (RLHF) and Constitutional AI (CAI) have been proposed for LLM alignment. However, these methods require either heavy human annotations or explicitly pre-defined constitutions, which are labor-intensive and resource-consuming. To overcome these drawbacks, we study constitution-based LLM alignment and propose a data-driven constitution discovery and self-alignment framework called IterAlign. IterAlign leverages red teaming to unveil the weaknesses of an LLM and automatically discovers new constitutions using a stronger LLM. These constitutions are then used to guide self-correction of the base LLM. Such a constitution discovery pipeline can be run iteratively and automatically to discover new constitutions that specifically target the alignment gaps in the current LLM. Empirical results on several safety benchmark datasets and multiple base LLMs show that IterAlign successfully improves truthfulness, helpfulness, harmlessness and honesty, improving the LLM alignment by up to $13.5\%$ in harmlessness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_18341 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | IterAlign: Iterative Constitutional Alignment of Large Language Models Chen, Xiusi Wen, Hongzhi Nag, Sreyashi Luo, Chen Yin, Qingyu Li, Ruirui Li, Zheng Wang, Wei Computation and Language With the rapid development of large language models (LLMs), aligning LLMs with human values and societal norms to ensure their reliability and safety has become crucial. Reinforcement learning with human feedback (RLHF) and Constitutional AI (CAI) have been proposed for LLM alignment. However, these methods require either heavy human annotations or explicitly pre-defined constitutions, which are labor-intensive and resource-consuming. To overcome these drawbacks, we study constitution-based LLM alignment and propose a data-driven constitution discovery and self-alignment framework called IterAlign. IterAlign leverages red teaming to unveil the weaknesses of an LLM and automatically discovers new constitutions using a stronger LLM. These constitutions are then used to guide self-correction of the base LLM. Such a constitution discovery pipeline can be run iteratively and automatically to discover new constitutions that specifically target the alignment gaps in the current LLM. Empirical results on several safety benchmark datasets and multiple base LLMs show that IterAlign successfully improves truthfulness, helpfulness, harmlessness and honesty, improving the LLM alignment by up to $13.5\%$ in harmlessness. |
| title | IterAlign: Iterative Constitutional Alignment of Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2403.18341 |