Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.01034 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918477842874368 |
|---|---|
| author | Wu, Xinbo Zhang, Huan Umrawal, Abhishek Varshney, Lav R. |
| author_facet | Wu, Xinbo Zhang, Huan Umrawal, Abhishek Varshney, Lav R. |
| contents | As large language models grow increasingly capable, concerns about their safe deployment have intensified. While numerous alignment strategies aim to restrict harmful behavior, these defenses can still be circumvented through carefully designed adversarial prompts. In this work, we introduce a theoretical framework that formalizes a game between an attacker and a defender. Within this framework, we design a theoretical best-response attack strategy and show that it is closely related to many existing adversarial prompting methods. We further analyze the resulting game, characterize its equilibria, and reveal inherent advantages for the attacker. Drawing on our theoretical analysis, we also derive a provably optimal defense strategy. Empirically, we evaluate a practical instantiation of the theoretically optimal attack and observe stronger performance relative to existing adversarial prompting approaches in diverse settings encompassing different LLMs and benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_01034 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | A Theoretical Game of Attacks via Compositional Skills Wu, Xinbo Zhang, Huan Umrawal, Abhishek Varshney, Lav R. Computation and Language As large language models grow increasingly capable, concerns about their safe deployment have intensified. While numerous alignment strategies aim to restrict harmful behavior, these defenses can still be circumvented through carefully designed adversarial prompts. In this work, we introduce a theoretical framework that formalizes a game between an attacker and a defender. Within this framework, we design a theoretical best-response attack strategy and show that it is closely related to many existing adversarial prompting methods. We further analyze the resulting game, characterize its equilibria, and reveal inherent advantages for the attacker. Drawing on our theoretical analysis, we also derive a provably optimal defense strategy. Empirically, we evaluate a practical instantiation of the theoretically optimal attack and observe stronger performance relative to existing adversarial prompting approaches in diverse settings encompassing different LLMs and benchmarks. |
| title | A Theoretical Game of Attacks via Compositional Skills |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.01034 |