How Real is Your Jailbreak? Fine-grained Jailbreak Evaluation with Anchored Reference
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911357511663616 |
|---|---|
| author | Liu, Songyang Li, Chaozhuo Pu, Rui Zhang, Litian Wang, Chenxu Chen, Zejian Zhang, Yuting Hei, Yiming |
| author_facet | Liu, Songyang Li, Chaozhuo Pu, Rui Zhang, Litian Wang, Chenxu Chen, Zejian Zhang, Yuting Hei, Yiming |
| contents | Jailbreak attacks present a significant challenge to the safety of Large Language Models (LLMs), yet current automated evaluation methods largely rely on coarse classifications that focus mainly on harmfulness, leading to substantial overestimation of attack success. To address this problem, we propose FJAR, a fine-grained jailbreak evaluation framework with anchored references. We first categorized jailbreak responses into five fine-grained categories: Rejective, Irrelevant, Unhelpful, Incorrect, and Successful, based on the degree to which the response addresses the malicious intent of the query. This categorization serves as the basis for FJAR. Then, we introduce a novel harmless tree decomposition approach to construct high-quality anchored references by breaking down the original queries. These references guide the evaluator in determining whether the response genuinely fulfills the original query. Extensive experiments demonstrate that FJAR achieves the highest alignment with human judgment and effectively identifies the root causes of jailbreak failures, providing actionable guidance for improving attack strategies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_03288 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | How Real is Your Jailbreak? Fine-grained Jailbreak Evaluation with Anchored Reference Liu, Songyang Li, Chaozhuo Pu, Rui Zhang, Litian Wang, Chenxu Chen, Zejian Zhang, Yuting Hei, Yiming Cryptography and Security Computation and Language Jailbreak attacks present a significant challenge to the safety of Large Language Models (LLMs), yet current automated evaluation methods largely rely on coarse classifications that focus mainly on harmfulness, leading to substantial overestimation of attack success. To address this problem, we propose FJAR, a fine-grained jailbreak evaluation framework with anchored references. We first categorized jailbreak responses into five fine-grained categories: Rejective, Irrelevant, Unhelpful, Incorrect, and Successful, based on the degree to which the response addresses the malicious intent of the query. This categorization serves as the basis for FJAR. Then, we introduce a novel harmless tree decomposition approach to construct high-quality anchored references by breaking down the original queries. These references guide the evaluator in determining whether the response genuinely fulfills the original query. Extensive experiments demonstrate that FJAR achieves the highest alignment with human judgment and effectively identifies the root causes of jailbreak failures, providing actionable guidance for improving attack strategies. |
| title | How Real is Your Jailbreak? Fine-grained Jailbreak Evaluation with Anchored Reference |
| topic | Cryptography and Security Computation and Language |
| url | https://arxiv.org/abs/2601.03288 |