Dishonesty in Helpful and Harmless Alignment
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916274290819072 |
|---|---|
| author | Huang, Youcheng Tang, Jingkun Feng, Duanyu Zhang, Zheng Lei, Wenqiang Lv, Jiancheng Cohn, Anthony G. |
| author_facet | Huang, Youcheng Tang, Jingkun Feng, Duanyu Zhang, Zheng Lei, Wenqiang Lv, Jiancheng Cohn, Anthony G. |
| contents | People tell lies when seeking rewards. Large language models (LLMs) are aligned to human values with reinforcement learning where they get rewards if they satisfy human preference. We find that this also induces dishonesty in helpful and harmless alignment where LLMs tell lies in generating harmless responses. Using the latest interpreting tools, we detect dishonesty, show how LLMs can be harmful if their honesty is increased, and analyze such conflicts at the parameter-level. Given these preliminaries and the hypothesis that reward-seeking stimulates dishonesty, we theoretically show that the dishonesty can in-turn decrease the alignment performances and augment reward-seeking alignment with representation regularization. Extensive results, including GPT-4 annotated win-rates, perplexities, and cases studies demonstrate that we can train more honest, helpful, and harmless LLMs. We will make all our codes and results be open-sourced upon this paper's acceptance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_01931 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Dishonesty in Helpful and Harmless Alignment Huang, Youcheng Tang, Jingkun Feng, Duanyu Zhang, Zheng Lei, Wenqiang Lv, Jiancheng Cohn, Anthony G. Computation and Language People tell lies when seeking rewards. Large language models (LLMs) are aligned to human values with reinforcement learning where they get rewards if they satisfy human preference. We find that this also induces dishonesty in helpful and harmless alignment where LLMs tell lies in generating harmless responses. Using the latest interpreting tools, we detect dishonesty, show how LLMs can be harmful if their honesty is increased, and analyze such conflicts at the parameter-level. Given these preliminaries and the hypothesis that reward-seeking stimulates dishonesty, we theoretically show that the dishonesty can in-turn decrease the alignment performances and augment reward-seeking alignment with representation regularization. Extensive results, including GPT-4 annotated win-rates, perplexities, and cases studies demonstrate that we can train more honest, helpful, and harmless LLMs. We will make all our codes and results be open-sourced upon this paper's acceptance. |
| title | Dishonesty in Helpful and Harmless Alignment |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.01931 |