LLM Hypnosis: Exploiting User Feedback for Unauthorized Knowledge Injection to All Users
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913046672179200 |
|---|---|
| author | Hilel, Almog Bhagwat, Riddhi Shenfeld, Idan Andreas, Jacob Choshen, Leshem |
| author_facet | Hilel, Almog Bhagwat, Riddhi Shenfeld, Idan Andreas, Jacob Choshen, Leshem |
| contents | We describe a vulnerability in language models (LMs) trained with user feedback, whereby a single user can persistently alter LM knowledge and behavior given only the ability to provide prompts and upvote / downvote feedback on LM outputs. To implement the attack, the attacker prompts the LM to stochastically output either a "poisoned" or benign response, then upvotes the poisoned response or downvotes the benign one. When feedback signals are used in a subsequent preference tuning behavior, LMs exhibit increased probability of producing poisoned responses even in contexts without malicious prompts. We show that this attack can be used to (1) insert factual knowledge the model did not previously possess, (2) modify code generation patterns in ways that introduce exploitable security flaws, and (3) inject fake financial news. Our finding both identifies a new qualitative feature of language model preference tuning (showing that it even highly restricted forms of preference data can be used to exert fine-grained control over behavior), and a new attack mechanism for LMs trained with user feedback (extending work on pretraining-time data poisoning and deployment-time prompt injection). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_02850 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LLM Hypnosis: Exploiting User Feedback for Unauthorized Knowledge Injection to All Users Hilel, Almog Bhagwat, Riddhi Shenfeld, Idan Andreas, Jacob Choshen, Leshem Computation and Language Cryptography and Security Machine Learning We describe a vulnerability in language models (LMs) trained with user feedback, whereby a single user can persistently alter LM knowledge and behavior given only the ability to provide prompts and upvote / downvote feedback on LM outputs. To implement the attack, the attacker prompts the LM to stochastically output either a "poisoned" or benign response, then upvotes the poisoned response or downvotes the benign one. When feedback signals are used in a subsequent preference tuning behavior, LMs exhibit increased probability of producing poisoned responses even in contexts without malicious prompts. We show that this attack can be used to (1) insert factual knowledge the model did not previously possess, (2) modify code generation patterns in ways that introduce exploitable security flaws, and (3) inject fake financial news. Our finding both identifies a new qualitative feature of language model preference tuning (showing that it even highly restricted forms of preference data can be used to exert fine-grained control over behavior), and a new attack mechanism for LMs trained with user feedback (extending work on pretraining-time data poisoning and deployment-time prompt injection). |
| title | LLM Hypnosis: Exploiting User Feedback for Unauthorized Knowledge Injection to All Users |
| topic | Computation and Language Cryptography and Security Machine Learning |
| url | https://arxiv.org/abs/2507.02850 |