RNR: Teaching Large Language Models to Follow Roles and Rules
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912040969306112 |
|---|---|
| author | Wang, Kuan Bukharin, Alexander Jiang, Haoming Yin, Qingyu Wang, Zhengyang Zhao, Tuo Shang, Jingbo Zhang, Chao Yin, Bing Li, Xian Chen, Jianshu Li, Shiyang |
| author_facet | Wang, Kuan Bukharin, Alexander Jiang, Haoming Yin, Qingyu Wang, Zhengyang Zhao, Tuo Shang, Jingbo Zhang, Chao Yin, Bing Li, Xian Chen, Jianshu Li, Shiyang |
| contents | Instruction fine-tuning (IFT) elicits instruction following capabilities and steers the behavior of large language models (LLMs) via supervised learning. However, existing models trained on open-source IFT datasets only have the ability to follow instructions from users, and often fail to follow complex role and rules specified by developers, a.k.a. system prompts. The ability to follow these roles and rules is essential for deployment, as it ensures that the model safely interacts with users within developer defined guidelines. To improve such role and rule following ability, we propose \model, an automated data generation pipeline that generates diverse roles and rules from existing IFT instructions, along with corresponding responses. This data can then be used to train models that follow complex system prompts. The models are evaluated on our newly created benchmarks for role and rule following ability, as well as standard instruction-following benchmarks and general NLP tasks. Our framework significantly improves role and rule following capability in LLMs, as evidenced by over 25% increase in pass-rate on rule adherence, i.e. following all requirements, in our experiments with the Alpaca and Ultrachat datasets. Moreover, our models achieves this increase without any regression on popular instruction following benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_13733 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | RNR: Teaching Large Language Models to Follow Roles and Rules Wang, Kuan Bukharin, Alexander Jiang, Haoming Yin, Qingyu Wang, Zhengyang Zhao, Tuo Shang, Jingbo Zhang, Chao Yin, Bing Li, Xian Chen, Jianshu Li, Shiyang Computation and Language Artificial Intelligence Human-Computer Interaction Instruction fine-tuning (IFT) elicits instruction following capabilities and steers the behavior of large language models (LLMs) via supervised learning. However, existing models trained on open-source IFT datasets only have the ability to follow instructions from users, and often fail to follow complex role and rules specified by developers, a.k.a. system prompts. The ability to follow these roles and rules is essential for deployment, as it ensures that the model safely interacts with users within developer defined guidelines. To improve such role and rule following ability, we propose \model, an automated data generation pipeline that generates diverse roles and rules from existing IFT instructions, along with corresponding responses. This data can then be used to train models that follow complex system prompts. The models are evaluated on our newly created benchmarks for role and rule following ability, as well as standard instruction-following benchmarks and general NLP tasks. Our framework significantly improves role and rule following capability in LLMs, as evidenced by over 25% increase in pass-rate on rule adherence, i.e. following all requirements, in our experiments with the Alpaca and Ultrachat datasets. Moreover, our models achieves this increase without any regression on popular instruction following benchmarks. |
| title | RNR: Teaching Large Language Models to Follow Roles and Rules |
| topic | Computation and Language Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2409.13733 |