Cover Image

Activation Steering Meets Preference Optimization: Defense Against Jailbreaks in Vision Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Wu, Sihao, Jin, Gaojie, Huang, Wei, Wang, Jianhong, Huang, Xiaowei
Format:	Preprint
Published:	2025
Subjects:	Computer Vision and Pattern Recognition Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks
by: Wang, Han, et al.
Published: (2024)

Adversarial Defense in Vision-Language Models: An Overview
by: Fu, Xiaowei, et al.
Published: (2026)

Defense-to-Attack: Bypassing Weak Defenses Enables Stronger Jailbreaks in Vision-Language Models
by: Zhao, Yunhan, et al.
Published: (2025)

TAIJI: Textual Anchoring for Immunizing Jailbreak Images in Vision Language Models
by: Yin, Xiangyu, et al.
Published: (2025)

White-box Multimodal Jailbreaks Against Large Vision-Language Models
by: Wang, Ruofan, et al.
Published: (2024)

Preference Alignment on Diffusion Model: A Comprehensive Survey for Image Generation and Editing
by: Wu, Sihao, et al.
Published: (2025)

Integrating Object Detection Modality into Visual Language Model for Enhanced Autonomous Driving Agent
by: He, Linfeng, et al.
Published: (2024)

Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models
by: Yin, Jianghao, et al.
Published: (2026)

Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attacks
by: Hossain, Md Zarif, et al.
Published: (2024)

Steering to Say No: Configurable Refusal via Activation Steering in Vision Language Models
by: Yang, Jiaxi, et al.
Published: (2026)

Toward Universal and Transferable Jailbreak Attacks on Vision-Language Models
by: Cui, Kaiyuan, et al.
Published: (2026)

Explanation-Driven Counterfactual Testing for Faithfulness in Vision-Language Model Explanations
by: Ding, Sihao, et al.
Published: (2025)

LPOI: Listwise Preference Optimization for Vision Language Models
by: Zadeh, Fatemeh Pesaran, et al.
Published: (2025)

IDEATOR: Jailbreaking and Benchmarking Large Vision-Language Models Using Themselves
by: Wang, Ruofan, et al.
Published: (2024)

BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attacks
by: Zhao, Yunhan, et al.
Published: (2024)

Jailbreaks on Vision Language Model via Multimodal Reasoning
by: Noheria, Aarush, et al.
Published: (2026)

Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimization
by: Wu, Jiulong, et al.
Published: (2025)

Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks
by: Zhou, Andy, et al.
Published: (2024)

Learning to Detect Unknown Jailbreak Attacks in Large Vision-Language Models
by: Liang, Shuang, et al.
Published: (2025)

ImgTrojan: Jailbreaking Vision-Language Models with ONE Image
by: Tao, Xijia, et al.
Published: (2024)

E$^2$AT: Multimodal Jailbreak Defense via Dynamic Joint Optimization for Multimodal Large Language Models
by: Lu, Liming, et al.
Published: (2025)

Mitigating Entangled Steering in Large Vision-Language Models for Hallucination Reduction
by: Zhang, Yuanhong, et al.
Published: (2026)

Visual Preference Optimization with Rubric Rewards
by: Yu, Ya-Qi, et al.
Published: (2026)

Underwater Camouflaged Object Tracking Meets Vision-Language SAM2
by: Zhang, Chunhui, et al.
Published: (2024)

Robust Vision-Language Models via Tensor Decomposition: A Defense Against Adversarial Attacks
by: Patel, Het, et al.
Published: (2025)

MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models
by: Liu, Ziyu, et al.
Published: (2024)

Principled Steering via Null-space Projection for Jailbreak Defense in Vision-Language Models
by: Zhu, Xingyu, et al.
Published: (2026)

AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimization
by: Liu, Chaohu, et al.
Published: (2025)

Rethinking Token Reduction for Large Vision-Language Models
by: Wang, Yi, et al.
Published: (2026)

V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization
by: Xie, Yuxi, et al.
Published: (2024)

AirCache: Activating Inter-modal Relevancy KV Cache Compression for Efficient Large Vision-Language Model Inference
by: Huang, Kai, et al.
Published: (2025)

ArchSIBench: Benchmarking the Architectural Spatial Intelligence of Vision-Language Models
by: Shen, Qirui, et al.
Published: (2026)

Jailbreaking Vision-Language Models Through the Visual Modality
by: Azulay, Aharon, et al.
Published: (2026)

VISOR++: Universal Visual Inputs based Steering for Large Vision Language Models
by: Balakrishnan, Ravikumar, et al.
Published: (2025)

VISOR: Visual Input-based Steering for Output Redirection in Vision-Language Models
by: Phute, Mansi, et al.
Published: (2025)

When the Prompt Becomes Visual: Vision-Centric Jailbreak Attacks for Large Image Editing Models
by: Hou, Jiacheng, et al.
Published: (2026)

Learning to Detect Unseen Jailbreak Attacks in Large Vision-Language Models
by: Liang, Shuang, et al.
Published: (2025)

Steering the Verifiability of Multimodal AI Hallucinations
by: Pang, Jianhong, et al.
Published: (2026)

Text is All You Need for Vision-Language Model Jailbreaking
by: Chen, Yihang, et al.
Published: (2026)

MedSteer: Counterfactual Endoscopic Synthesis via Training-Free Activation Steering
by: Pham, Trong-Thang, et al.
Published: (2026)