PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915342329053184 |
|---|---|
| author | Ji, Jiaming Hong, Donghai Zhang, Borong Chen, Boyuan Dai, Juntao Zheng, Boren Qiu, Tianyi Zhou, Jiayi Wang, Kaile Li, Boxuan Han, Sirui Guo, Yike Yang, Yaodong |
| author_facet | Ji, Jiaming Hong, Donghai Zhang, Borong Chen, Boyuan Dai, Juntao Zheng, Boren Qiu, Tianyi Zhou, Jiayi Wang, Kaile Li, Boxuan Han, Sirui Guo, Yike Yang, Yaodong |
| contents | In this study, we introduce the safety human preference dataset, PKU-SafeRLHF, designed to promote research on safety alignment in large language models (LLMs). As a sibling project to SafeRLHF and BeaverTails, we separate annotations of helpfulness and harmlessness for question-answering pairs, providing distinct perspectives on these coupled attributes. Overall, we provide 44.6k refined prompts and 265k question-answer pairs with safety meta-labels for 19 harm categories and three severity levels ranging from minor to severe, with answers generated by Llama-family models. Based on this, we collected 166.8k preference data, including dual-preference (helpfulness and harmlessness decoupled) and single-preference data (trade-off the helpfulness and harmlessness from scratch), respectively. Using the large-scale annotation data, we further train severity-sensitive moderation for the risk control of LLMs and safety-centric RLHF algorithms for the safety alignment of LLMs. We believe this dataset will be a valuable resource for the community, aiding in the safe deployment of LLMs. Data is available at https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_15513 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference Ji, Jiaming Hong, Donghai Zhang, Borong Chen, Boyuan Dai, Juntao Zheng, Boren Qiu, Tianyi Zhou, Jiayi Wang, Kaile Li, Boxuan Han, Sirui Guo, Yike Yang, Yaodong Artificial Intelligence Computation and Language In this study, we introduce the safety human preference dataset, PKU-SafeRLHF, designed to promote research on safety alignment in large language models (LLMs). As a sibling project to SafeRLHF and BeaverTails, we separate annotations of helpfulness and harmlessness for question-answering pairs, providing distinct perspectives on these coupled attributes. Overall, we provide 44.6k refined prompts and 265k question-answer pairs with safety meta-labels for 19 harm categories and three severity levels ranging from minor to severe, with answers generated by Llama-family models. Based on this, we collected 166.8k preference data, including dual-preference (helpfulness and harmlessness decoupled) and single-preference data (trade-off the helpfulness and harmlessness from scratch), respectively. Using the large-scale annotation data, we further train severity-sensitive moderation for the risk control of LLMs and safety-centric RLHF algorithms for the safety alignment of LLMs. We believe this dataset will be a valuable resource for the community, aiding in the safe deployment of LLMs. Data is available at https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF. |
| title | PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2406.15513 |