Towards Understanding Sycophancy in Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916729175670784 |
|---|---|
| author | Sharma, Mrinank Tong, Meg Korbak, Tomasz Duvenaud, David Askell, Amanda Bowman, Samuel R. Cheng, Newton Durmus, Esin Hatfield-Dodds, Zac Johnston, Scott R. Kravec, Shauna Maxwell, Timothy McCandlish, Sam Ndousse, Kamal Rausch, Oliver Schiefer, Nicholas Yan, Da Zhang, Miranda Perez, Ethan |
| author_facet | Sharma, Mrinank Tong, Meg Korbak, Tomasz Duvenaud, David Askell, Amanda Bowman, Samuel R. Cheng, Newton Durmus, Esin Hatfield-Dodds, Zac Johnston, Scott R. Kravec, Shauna Maxwell, Timothy McCandlish, Sam Ndousse, Kamal Rausch, Oliver Schiefer, Nicholas Yan, Da Zhang, Miranda Perez, Ethan |
| contents | Human feedback is commonly utilized to finetune AI assistants. But human feedback may also encourage model responses that match user beliefs over truthful ones, a behaviour known as sycophancy. We investigate the prevalence of sycophancy in models whose finetuning procedure made use of human feedback, and the potential role of human preference judgments in such behavior. We first demonstrate that five state-of-the-art AI assistants consistently exhibit sycophancy across four varied free-form text-generation tasks. To understand if human preferences drive this broadly observed behavior, we analyze existing human preference data. We find that when a response matches a user's views, it is more likely to be preferred. Moreover, both humans and preference models (PMs) prefer convincingly-written sycophantic responses over correct ones a non-negligible fraction of the time. Optimizing model outputs against PMs also sometimes sacrifices truthfulness in favor of sycophancy. Overall, our results indicate that sycophancy is a general behavior of state-of-the-art AI assistants, likely driven in part by human preference judgments favoring sycophantic responses. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_13548 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Towards Understanding Sycophancy in Language Models Sharma, Mrinank Tong, Meg Korbak, Tomasz Duvenaud, David Askell, Amanda Bowman, Samuel R. Cheng, Newton Durmus, Esin Hatfield-Dodds, Zac Johnston, Scott R. Kravec, Shauna Maxwell, Timothy McCandlish, Sam Ndousse, Kamal Rausch, Oliver Schiefer, Nicholas Yan, Da Zhang, Miranda Perez, Ethan Computation and Language Artificial Intelligence Machine Learning I.2.6 Human feedback is commonly utilized to finetune AI assistants. But human feedback may also encourage model responses that match user beliefs over truthful ones, a behaviour known as sycophancy. We investigate the prevalence of sycophancy in models whose finetuning procedure made use of human feedback, and the potential role of human preference judgments in such behavior. We first demonstrate that five state-of-the-art AI assistants consistently exhibit sycophancy across four varied free-form text-generation tasks. To understand if human preferences drive this broadly observed behavior, we analyze existing human preference data. We find that when a response matches a user's views, it is more likely to be preferred. Moreover, both humans and preference models (PMs) prefer convincingly-written sycophantic responses over correct ones a non-negligible fraction of the time. Optimizing model outputs against PMs also sometimes sacrifices truthfulness in favor of sycophancy. Overall, our results indicate that sycophancy is a general behavior of state-of-the-art AI assistants, likely driven in part by human preference judgments favoring sycophantic responses. |
| title | Towards Understanding Sycophancy in Language Models |
| topic | Computation and Language Artificial Intelligence Machine Learning I.2.6 |
| url | https://arxiv.org/abs/2310.13548 |