Enhancing Diffusion-based Unrestricted Adversarial Attacks via Adversary Preferences Alignment
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916772684234752 |
|---|---|
| author | Jiang, Kaixun Chen, Zhaoyu Guo, Haijing Li, Jinglun Fu, Jiyuan Guo, Pinxue Tang, Hao Li, Bo Zhang, Wenqiang |
| author_facet | Jiang, Kaixun Chen, Zhaoyu Guo, Haijing Li, Jinglun Fu, Jiyuan Guo, Pinxue Tang, Hao Li, Bo Zhang, Wenqiang |
| contents | Preference alignment in diffusion models has primarily focused on benign human preferences (e.g., aesthetic). In this paper, we propose a novel perspective: framing unrestricted adversarial example generation as a problem of aligning with adversary preferences. Unlike benign alignment, adversarial alignment involves two inherently conflicting preferences: visual consistency and attack effectiveness, which often lead to unstable optimization and reward hacking (e.g., reducing visual quality to improve attack success). To address this, we propose APA (Adversary Preferences Alignment), a two-stage framework that decouples conflicting preferences and optimizes each with differentiable rewards. In the first stage, APA fine-tunes LoRA to improve visual consistency using rule-based similarity reward. In the second stage, APA updates either the image latent or prompt embedding based on feedback from a substitute classifier, guided by trajectory-level and step-wise rewards. To enhance black-box transferability, we further incorporate a diffusion augmentation strategy. Experiments demonstrate that APA achieves significantly better attack transferability while maintaining high visual consistency, inspiring further research to approach adversarial attacks from an alignment perspective. Code will be available at https://github.com/deep-kaixun/APA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_01511 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Enhancing Diffusion-based Unrestricted Adversarial Attacks via Adversary Preferences Alignment Jiang, Kaixun Chen, Zhaoyu Guo, Haijing Li, Jinglun Fu, Jiyuan Guo, Pinxue Tang, Hao Li, Bo Zhang, Wenqiang Computer Vision and Pattern Recognition Preference alignment in diffusion models has primarily focused on benign human preferences (e.g., aesthetic). In this paper, we propose a novel perspective: framing unrestricted adversarial example generation as a problem of aligning with adversary preferences. Unlike benign alignment, adversarial alignment involves two inherently conflicting preferences: visual consistency and attack effectiveness, which often lead to unstable optimization and reward hacking (e.g., reducing visual quality to improve attack success). To address this, we propose APA (Adversary Preferences Alignment), a two-stage framework that decouples conflicting preferences and optimizes each with differentiable rewards. In the first stage, APA fine-tunes LoRA to improve visual consistency using rule-based similarity reward. In the second stage, APA updates either the image latent or prompt embedding based on feedback from a substitute classifier, guided by trajectory-level and step-wise rewards. To enhance black-box transferability, we further incorporate a diffusion augmentation strategy. Experiments demonstrate that APA achieves significantly better attack transferability while maintaining high visual consistency, inspiring further research to approach adversarial attacks from an alignment perspective. Code will be available at https://github.com/deep-kaixun/APA. |
| title | Enhancing Diffusion-based Unrestricted Adversarial Attacks via Adversary Preferences Alignment |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2506.01511 |