Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.21417 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909274957938688 |
|---|---|
| author | Wu, Zhengxuan Zhang, Yuhao Qi, Peng Xu, Yumo Han, Rujun Zhang, Yian Chen, Jifan Min, Bonan Huang, Zhiheng |
| author_facet | Wu, Zhengxuan Zhang, Yuhao Qi, Peng Xu, Yumo Han, Rujun Zhang, Yian Chen, Jifan Min, Bonan Huang, Zhiheng |
| contents | Modern language models (LMs) need to follow human instructions while being faithful; yet, they often fail to achieve both. Here, we provide concrete evidence of a trade-off between instruction following (i.e., follow open-ended instructions) and faithfulness (i.e., ground responses in given context) when training LMs with these objectives. For instance, fine-tuning LLaMA-7B on instruction following datasets renders it less faithful. Conversely, instruction-tuned Vicuna-7B shows degraded performance at following instructions when further optimized on tasks that require contextual grounding. One common remedy is multi-task learning (MTL) with data mixing, yet it remains far from achieving a synergic outcome. We propose a simple yet effective method that relies on Rejection Sampling for Continued Self-instruction Tuning (ReSet), which significantly outperforms vanilla MTL. Surprisingly, we find that less is more, as training ReSet with high-quality, yet substantially smaller data (three-fold less) yields superior results. Our findings offer a better understanding of objective discrepancies in alignment training of LMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_21417 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models Wu, Zhengxuan Zhang, Yuhao Qi, Peng Xu, Yumo Han, Rujun Zhang, Yian Chen, Jifan Min, Bonan Huang, Zhiheng Computation and Language Modern language models (LMs) need to follow human instructions while being faithful; yet, they often fail to achieve both. Here, we provide concrete evidence of a trade-off between instruction following (i.e., follow open-ended instructions) and faithfulness (i.e., ground responses in given context) when training LMs with these objectives. For instance, fine-tuning LLaMA-7B on instruction following datasets renders it less faithful. Conversely, instruction-tuned Vicuna-7B shows degraded performance at following instructions when further optimized on tasks that require contextual grounding. One common remedy is multi-task learning (MTL) with data mixing, yet it remains far from achieving a synergic outcome. We propose a simple yet effective method that relies on Rejection Sampling for Continued Self-instruction Tuning (ReSet), which significantly outperforms vanilla MTL. Surprisingly, we find that less is more, as training ReSet with high-quality, yet substantially smaller data (three-fold less) yields superior results. Our findings offer a better understanding of objective discrepancies in alignment training of LMs. |
| title | Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2407.21417 |