Copertina

Learning To See But Forgetting To Follow: Visual Instruction Tuning Makes LLMs More Prone To Jailbreak Attacks

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Pantazopoulos, Georgios, Parekh, Amit, Nikandrou, Malvina, Suglia, Alessandro
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Computer Vision and Pattern Recognition Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Enhancing Continual Learning in Visual Question Answering with Modality-Aware Feature Distillation
di: Nikandrou, Malvina, et al.
Pubblicazione: (2024)

CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Concepts
di: Nikandrou, Malvina, et al.
Pubblicazione: (2024)

Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modeling
di: Pantazopoulos, Georgios, et al.
Pubblicazione: (2024)

Retrievit: In-context Retrieval Capabilities of Transformers, State Space Models, and Hybrid Architectures
di: Pantazopoulos, Georgios, et al.
Pubblicazione: (2026)

Lost in Space: Probing Fine-grained Spatial Understanding in Vision and Language Resamplers
di: Pantazopoulos, Georgios, et al.
Pubblicazione: (2024)

AgriPath: A Systematic Exploration of Architectural Trade-offs for Crop Disease Classification
di: Mooraj, Hamza, et al.
Pubblicazione: (2026)

What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning
di: Du, Yifan, et al.
Pubblicazione: (2023)

Less is More: High-value Data Selection for Visual Instruction Tuning
di: Liu, Zikang, et al.
Pubblicazione: (2024)

Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Users
di: Karamolegkou, Antonia, et al.
Pubblicazione: (2025)

Investigating the Role of Instruction Variety and Task Difficulty in Robotic Manipulation Tasks
di: Parekh, Amit, et al.
Pubblicazione: (2024)

FOSSIL: Harnessing Feedback on Suboptimal Samples for Data-Efficient Generalisation with Imitation Learning for Embodied Vision-and-Language Tasks
di: McCallum, Sabrina, et al.
Pubblicazione: (2025)

Towards Understanding Visual Grounding in Visual Language Models
di: Pantazopoulos, Georgios, et al.
Pubblicazione: (2025)

Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context
di: Zheng, Ge, et al.
Pubblicazione: (2025)

MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs
di: Qian, Yusu, et al.
Pubblicazione: (2024)

Learning to Instruct for Visual Instruction Tuning
di: Zhou, Zhihan, et al.
Pubblicazione: (2025)

Visual Contextual Attack: Jailbreaking MLLMs with Image-Driven Context Injection
di: Miao, Ziqi, et al.
Pubblicazione: (2025)

Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning
di: Xu, Zhiyang, et al.
Pubblicazione: (2024)

LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning
di: Cocchi, Federico, et al.
Pubblicazione: (2025)

LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding
di: Zhang, Yanzhe, et al.
Pubblicazione: (2023)

Reconstructive Visual Instruction Tuning
di: Wang, Haochen, et al.
Pubblicazione: (2024)

Cross-Modal Obfuscation for Jailbreak Attacks on Large Vision-Language Models
di: Jiang, Lei, et al.
Pubblicazione: (2025)

Jailbreak Attacks and Defenses against Multimodal Generative Models: A Survey
di: Liu, Xuannan, et al.
Pubblicazione: (2024)

Instruction Makes a Difference
di: Adewumi, Tosin, et al.
Pubblicazione: (2024)

One RL to See Them All: Visual Triple Unified Reinforcement Learning
di: Ma, Yan, et al.
Pubblicazione: (2025)

See the Text: From Tokenization to Visual Reading
di: Xing, Ling, et al.
Pubblicazione: (2025)

Improved Baselines with Visual Instruction Tuning
di: Liu, Haotian, et al.
Pubblicazione: (2023)

Parrot: Multilingual Visual Instruction Tuning
di: Sun, Hai-Long, et al.
Pubblicazione: (2024)

Automatic Layout Planning for Visually-Rich Documents with Instruction-Following Models
di: Zhu, Wanrong, et al.
Pubblicazione: (2024)

AMIA: Automatic Masking and Joint Intention Analysis Makes LVLMs Robust Jailbreak Defenders
di: Zhang, Yuqi, et al.
Pubblicazione: (2025)

Context-aware Visual Storytelling with Visual Prefix Tuning and Contrastive Learning
di: Song, Yingjin, et al.
Pubblicazione: (2024)

Instruction-Following Evaluation of Large Vision-Language Models
di: Shiono, Daiki, et al.
Pubblicazione: (2025)

LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering
di: Bi, Jinhe, et al.
Pubblicazione: (2024)

LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning
di: You, Zebin, et al.
Pubblicazione: (2025)

Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination
di: Shi, Chufan, et al.
Pubblicazione: (2026)

Does Instruction Tuning Make LLMs More Consistent?
di: Fierro, Constanza, et al.
Pubblicazione: (2024)

Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking
di: Li, Jingru, et al.
Pubblicazione: (2026)

Sharing the Cost of Success: A Game for Evaluating and Learning Collaborative Multi-Agent Instruction Giving and Following Policies
di: Sadler, Philipp, et al.
Pubblicazione: (2024)

Analyzing Finetuning Representation Shift for Multimodal LLMs Steering
di: Khayatan, Pegah, et al.
Pubblicazione: (2025)

Can Visual Encoder Learn to See Arrows?
di: Terashita, Naoyuki, et al.
Pubblicazione: (2025)

Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models
di: Li, Yifan, et al.
Pubblicazione: (2024)