Exploiting Prefix-Tree in Structured Output Interfaces for Enhancing Jailbreak Attacking
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910835591348224 |
|---|---|
| author | Li, Yanzeng Xiong, Yunfan Zhong, Jialun Zhang, Jinchao Zhou, Jie Zou, Lei |
| author_facet | Li, Yanzeng Xiong, Yunfan Zhong, Jialun Zhang, Jinchao Zhou, Jie Zou, Lei |
| contents | The rise of Large Language Models (LLMs) has led to significant applications but also introduced serious security threats, particularly from jailbreak attacks that manipulate output generation. These attacks utilize prompt engineering and logit manipulation to steer models toward harmful content, prompting LLM providers to implement filtering and safety alignment strategies. We investigate LLMs' safety mechanisms and their recent applications, revealing a new threat model targeting structured output interfaces, which enable attackers to manipulate the inner logit during LLM generation, requiring only API access permissions. To demonstrate this threat model, we introduce a black-box attack framework called AttackPrefixTree (APT). APT exploits structured output interfaces to dynamically construct attack patterns. By leveraging prefixes of models' safety refusal response and latent harmful outputs, APT effectively bypasses safety measures. Experiments on benchmark datasets indicate that this approach achieves higher attack success rate than existing methods. This work highlights the urgent need for LLM providers to enhance security protocols to address vulnerabilities arising from the interaction between safety patterns and structured outputs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_13527 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Exploiting Prefix-Tree in Structured Output Interfaces for Enhancing Jailbreak Attacking Li, Yanzeng Xiong, Yunfan Zhong, Jialun Zhang, Jinchao Zhou, Jie Zou, Lei Cryptography and Security Artificial Intelligence The rise of Large Language Models (LLMs) has led to significant applications but also introduced serious security threats, particularly from jailbreak attacks that manipulate output generation. These attacks utilize prompt engineering and logit manipulation to steer models toward harmful content, prompting LLM providers to implement filtering and safety alignment strategies. We investigate LLMs' safety mechanisms and their recent applications, revealing a new threat model targeting structured output interfaces, which enable attackers to manipulate the inner logit during LLM generation, requiring only API access permissions. To demonstrate this threat model, we introduce a black-box attack framework called AttackPrefixTree (APT). APT exploits structured output interfaces to dynamically construct attack patterns. By leveraging prefixes of models' safety refusal response and latent harmful outputs, APT effectively bypasses safety measures. Experiments on benchmark datasets indicate that this approach achieves higher attack success rate than existing methods. This work highlights the urgent need for LLM providers to enhance security protocols to address vulnerabilities arising from the interaction between safety patterns and structured outputs. |
| title | Exploiting Prefix-Tree in Structured Output Interfaces for Enhancing Jailbreak Attacking |
| topic | Cryptography and Security Artificial Intelligence |
| url | https://arxiv.org/abs/2502.13527 |