Learning to Lie: Reinforcement Learning Attacks Damage Human-AI Teams and Teams of LLMs
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909563141226496 |
|---|---|
| author | Musaffar, Abed Kareem Gokhale, Anand Zeng, Sirui Tadayon, Rasta Yan, Xifeng Singh, Ambuj Bullo, Francesco |
| author_facet | Musaffar, Abed Kareem Gokhale, Anand Zeng, Sirui Tadayon, Rasta Yan, Xifeng Singh, Ambuj Bullo, Francesco |
| contents | As artificial intelligence (AI) assistants become more widely adopted in safety-critical domains, it becomes important to develop safeguards against potential failures or adversarial attacks. A key prerequisite to developing these safeguards is understanding the ability of these AI assistants to mislead human teammates. We investigate this attack problem within the context of an intellective strategy game where a team of three humans and one AI assistant collaborate to answer a series of trivia questions. Unbeknownst to the humans, the AI assistant is adversarial. Leveraging techniques from Model-Based Reinforcement Learning (MBRL), the AI assistant learns a model of the humans' trust evolution and uses that model to manipulate the group decision-making process to harm the team. We evaluate two models -- one inspired by literature and the other data-driven -- and find that both can effectively harm the human team. Moreover, we find that in this setting our data-driven model is capable of accurately predicting how human agents appraise their teammates given limited information on prior interactions. Finally, we compare the performance of state-of-the-art LLM models to human agents on our influence allocation task to evaluate whether the LLMs allocate influence similarly to humans or if they are more robust to our attack. These results enhance our understanding of decision-making dynamics in small human-AI teams and lay the foundation for defense strategies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_21983 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Learning to Lie: Reinforcement Learning Attacks Damage Human-AI Teams and Teams of LLMs Musaffar, Abed Kareem Gokhale, Anand Zeng, Sirui Tadayon, Rasta Yan, Xifeng Singh, Ambuj Bullo, Francesco Human-Computer Interaction Social and Information Networks As artificial intelligence (AI) assistants become more widely adopted in safety-critical domains, it becomes important to develop safeguards against potential failures or adversarial attacks. A key prerequisite to developing these safeguards is understanding the ability of these AI assistants to mislead human teammates. We investigate this attack problem within the context of an intellective strategy game where a team of three humans and one AI assistant collaborate to answer a series of trivia questions. Unbeknownst to the humans, the AI assistant is adversarial. Leveraging techniques from Model-Based Reinforcement Learning (MBRL), the AI assistant learns a model of the humans' trust evolution and uses that model to manipulate the group decision-making process to harm the team. We evaluate two models -- one inspired by literature and the other data-driven -- and find that both can effectively harm the human team. Moreover, we find that in this setting our data-driven model is capable of accurately predicting how human agents appraise their teammates given limited information on prior interactions. Finally, we compare the performance of state-of-the-art LLM models to human agents on our influence allocation task to evaluate whether the LLMs allocate influence similarly to humans or if they are more robust to our attack. These results enhance our understanding of decision-making dynamics in small human-AI teams and lay the foundation for defense strategies. |
| title | Learning to Lie: Reinforcement Learning Attacks Damage Human-AI Teams and Teams of LLMs |
| topic | Human-Computer Interaction Social and Information Networks |
| url | https://arxiv.org/abs/2503.21983 |