SoFA: Shielded On-the-fly Alignment via Priority Rule Following
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917599229509632 |
|---|---|
| author | Lu, Xinyu Yu, Bowen Lu, Yaojie Lin, Hongyu Yu, Haiyang Sun, Le Han, Xianpei Li, Yongbin |
| author_facet | Lu, Xinyu Yu, Bowen Lu, Yaojie Lin, Hongyu Yu, Haiyang Sun, Le Han, Xianpei Li, Yongbin |
| contents | The alignment problem in Large Language Models (LLMs) involves adapting them to the broad spectrum of human values. This requirement challenges existing alignment methods due to diversity of preferences and regulatory standards. This paper introduces a novel alignment paradigm, priority rule following, which defines rules as the primary control mechanism in each dialog, prioritizing them over user instructions. Our preliminary analysis reveals that even the advanced LLMs, such as GPT-4, exhibit shortcomings in understanding and prioritizing the rules. Therefore, we present PriorityDistill, a semi-automated approach for distilling priority following signals from LLM simulations to ensure robust rule integration and adherence. Our experiments show that this method not only effectively minimizes misalignments utilizing only one general rule but also adapts smoothly to various unseen rules, ensuring they are shielded from hijacking and that the model responds appropriately. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_17358 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SoFA: Shielded On-the-fly Alignment via Priority Rule Following Lu, Xinyu Yu, Bowen Lu, Yaojie Lin, Hongyu Yu, Haiyang Sun, Le Han, Xianpei Li, Yongbin Computation and Language The alignment problem in Large Language Models (LLMs) involves adapting them to the broad spectrum of human values. This requirement challenges existing alignment methods due to diversity of preferences and regulatory standards. This paper introduces a novel alignment paradigm, priority rule following, which defines rules as the primary control mechanism in each dialog, prioritizing them over user instructions. Our preliminary analysis reveals that even the advanced LLMs, such as GPT-4, exhibit shortcomings in understanding and prioritizing the rules. Therefore, we present PriorityDistill, a semi-automated approach for distilling priority following signals from LLM simulations to ensure robust rule integration and adherence. Our experiments show that this method not only effectively minimizes misalignments utilizing only one general rule but also adapts smoothly to various unseen rules, ensuring they are shielded from hijacking and that the model responds appropriately. |
| title | SoFA: Shielded On-the-fly Alignment via Priority Rule Following |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.17358 |