Cover Image

Steering to Say No: Configurable Refusal via Activation Steering in Vision Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Yang, Jiaxi, Liu, Shicheng, Yang, Yuchen, Lee, Dongwon
Format:	Preprint
Published:	2026
Subjects:	Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Reducing Hallucinations in Vision-Language Models via Latent Space Steering
by: Liu, Sheng, et al.
Published: (2024)

The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steering
by: Li, Zhuowei, et al.
Published: (2025)

SAVeS: Steering Safety Judgments in Vision-Language Models via Semantic Cues
by: Hinojosa, Carlos, et al.
Published: (2026)

Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision-Language Models
by: Dafnis, Konstantinos M., et al.
Published: (2025)

Steering Sparse Autoencoder Latents to Control Dynamic Head Pruning in Vision Transformers (Student Abstract)
by: Lee, Yousung, et al.
Published: (2026)

SteerVLM: Robust Model Control through Lightweight Activation Steering for Vision Language Models
by: Sivakumar, Anushka, et al.
Published: (2025)

Learning to Steer: Input-dependent Steering for Multimodal LLMs
by: Parekh, Jayneel, et al.
Published: (2025)

Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws
by: Wei, Xiyuan, et al.
Published: (2025)

TraSCE: Trajectory Steering for Concept Erasure
by: Jain, Anubhav, et al.
Published: (2024)

Steering CLIP's vision transformer with sparse autoencoders
by: Joseph, Sonia, et al.
Published: (2025)

Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models
by: Yin, Jianghao, et al.
Published: (2026)

Steering LVLMs via Sparse Autoencoder for Hallucination Mitigation
by: Hua, Zhenglin, et al.
Published: (2025)

Learning What Matters: Steering Diffusion via Spectrally Anisotropic Forward Noise
by: Scimeca, Luca, et al.
Published: (2025)

SwordBench: Evaluating Orthogonality of Steering Image Representations
by: Zaigrajew, Vladimir, et al.
Published: (2026)

ROCKET-2: Steering Visuomotor Policy via Cross-View Goal Alignment
by: Cai, Shaofei, et al.
Published: (2025)

MedSteer: Counterfactual Endoscopic Synthesis via Training-Free Activation Steering
by: Pham, Trong-Thang, et al.
Published: (2026)

ATHENA: Adaptive Test-Time Steering for Improving Count Fidelity in Diffusion Models
by: Sepehri, Mohammad Shahab, et al.
Published: (2026)

Programming Refusal with Conditional Activation Steering
by: Lee, Bruce W., et al.
Published: (2024)

Steering Away from Memorization: Reachability-Constrained Reinforcement Learning for Text-to-Image Diffusion
by: Karnik, Sathwik, et al.
Published: (2026)

Activation Steering Meets Preference Optimization: Defense Against Jailbreaks in Vision Language Models
by: Wu, Sihao, et al.
Published: (2025)

Steering the Verifiability of Multimodal AI Hallucinations
by: Pang, Jianhong, et al.
Published: (2026)

AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition
by: Lin, Zichuan, et al.
Published: (2025)

Activation Matters: Test-time Activated Negative Labels for OOD Detection with Vision-Language Models
by: Zhang, Yabin, et al.
Published: (2026)

StressDream: Steering Video World Models for Robust Policy Evaluation and Improvement
by: Seo, Junwon, et al.
Published: (2026)

Steering the LoCoMotif: Using Domain Knowledge in Time Series Motif Discovery
by: Yurtman, Aras, et al.
Published: (2025)

Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steering
by: Liu, Shuliang, et al.
Published: (2026)

Refusing Safe Prompts for Multi-modal Large Language Models
by: Shao, Zedian, et al.
Published: (2024)

Beyond Interpretability: When, Why, and How Sparse Autoencoders Enable Label-Free Visual Steering
by: Chatzoudis, Gerasimos, et al.
Published: (2025)

Concept-Guided Fine-Tuning: Steering ViTs away from Spurious Correlations to Improve Robustness
by: Elisha, Yehonatan, et al.
Published: (2026)

Mitigating Entangled Steering in Large Vision-Language Models for Hallucination Reduction
by: Zhang, Yuanhong, et al.
Published: (2026)

Why Steering Works: Toward a Unified View of Language Model Parameter Dynamics
by: Xu, Ziwen, et al.
Published: (2026)

Vision-Language Models Can Self-Improve Reasoning via Reflection
by: Cheng, Kanzhi, et al.
Published: (2024)

Rethinking Post-Unlearning Behavior of Large Vision-Language Models
by: Kim, Minsung, et al.
Published: (2025)

LangGap: Diagnosing and Closing the Language Gap in Vision-Language-Action Models
by: Hou, Yuchen, et al.
Published: (2026)

VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning
by: Yang, Senqiao, et al.
Published: (2025)

EasyEdit2: An Easy-to-use Steering Framework for Editing Large Language Models
by: Xu, Ziwen, et al.
Published: (2025)

VLM in a flash: I/O-Efficient Sparsification of Vision-Language Model via Neuron Chunking
by: Yang, Kichang, et al.
Published: (2025)

VISOR++: Universal Visual Inputs based Steering for Large Vision Language Models
by: Balakrishnan, Ravikumar, et al.
Published: (2025)

VISOR: Visual Input-based Steering for Output Redirection in Vision-Language Models
by: Phute, Mansi, et al.
Published: (2025)

Language Models Can Explain Visual Features via Steering
by: Ferrando, Javier, et al.
Published: (2026)