Cover Image

Bootstrapping LLM Robustness for VLM Safety via Reducing the Pretraining Modality Gap

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Yang, Wenhan, Stice, Spencer, Payani, Ali, Mirzasoleiman, Baharan
Format:	Preprint
Published:	2025
Subjects:	Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

LoRA is All You Need for Safety Alignment of Reasoning LLMs
by: Xue, Yihao, et al.
Published: (2025)

Better Safe than Sorry: Pre-training CLIP against Targeted Data Poisoning and Backdoor Attacks
by: Yang, Wenhan, et al.
Published: (2023)

Tuning the Implicit Regularizer of Masked Diffusion Language Models: Enhancing Generalization via Insights from $k$-Parity
by: Huang, Jianhao, et al.
Published: (2026)

Data-Efficient Contrastive Self-supervised Learning: Most Beneficial Examples for Supervised Learning Contribute the Least
by: Joshi, Siddharth, et al.
Published: (2023)

Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity
by: Nguyen, Dang, et al.
Published: (2025)

Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixtures
by: Nguyen, Dang, et al.
Published: (2024)

Graph Contrastive Learning under Heterophily via Graph Filters
by: Yang, Wenhan, et al.
Published: (2023)

Data-Efficient Contrastive Language-Image Pretraining: Prioritizing Data Quality over Quantity
by: Joshi, Siddharth, et al.
Published: (2024)

Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictions
by: Xue, Yihao, et al.
Published: (2025)

Understanding the Role of Training Data in Test-Time Scaling
by: Javanmard, Adel, et al.
Published: (2025)

Theoretical Perspectives on Data Quality and Synergistic Effects in Pre- and Post-Training Reasoning Models
by: Javanmard, Adel, et al.
Published: (2026)

Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalization
by: Nguyen, Dang, et al.
Published: (2024)

Few-shot Adaptation to Distribution Shifts By Mixing Source and Target Embeddings
by: Xue, Yihao, et al.
Published: (2023)

Verify when Uncertain: Beyond Self-Consistency in Black Box Hallucination Detection
by: Xue, Yihao, et al.
Published: (2025)

SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models
by: Yang, Yu, et al.
Published: (2024)

How Transformers Learn to Plan via Multi-Token Prediction
by: Huang, Jianhao, et al.
Published: (2026)

Data Distribution as a Lever for Guiding Optimizers Toward Superior Generalization in LLMs
by: Gangavarapu, Tushaar, et al.
Published: (2026)

VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gap
by: Liu, Qin, et al.
Published: (2025)

PRISM: Robust VLM Alignment with Principled Reasoning for Integrated Safety in Multimodality
by: Li, Nanxi, et al.
Published: (2025)

Enhancing Foundation VLM Robustness to Missing Modality: Scalable Diffusion for Bi-directional Feature Restoration
by: Dai, Wei, et al.
Published: (2026)

Challenges and Opportunities in Improving Worst-Group Generalization in Presence of Spurious Features
by: Joshi, Siddharth, et al.
Published: (2023)

Attention Bootstrapping for Multi-Modal Test-Time Adaptation
by: Zhao, Yusheng, et al.
Published: (2025)

SafeCoT: Improving VLM Safety with Minimal Reasoning
by: Ma, Jiachen, et al.
Published: (2025)

Beyond Sharp Minima: Robust LLM Unlearning via Feedback-Guided Multi-Point Optimization
by: Wu, Wenhan, et al.
Published: (2025)

LAMB: LLM-based Audio Captioning with Modality Gap Bridging via Cauchy-Schwarz Divergence
by: Lee, Hyeongkeun, et al.
Published: (2026)

Bridging the Safety Gap: A Guardrail Pipeline for Trustworthy LLM Inferences
by: Han, Shanshan, et al.
Published: (2025)

Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization
by: Liu, Zhihao, et al.
Published: (2026)

When is Tree Search Useful for LLM Planning? It Depends on the Discriminator
by: Chen, Ziru, et al.
Published: (2024)

On the Vulnerability of LLM/VLM-Controlled Robotics
by: Wu, Xiyang, et al.
Published: (2024)

7B Fully Open Source Moxin-LLM/VLM -- From Pretraining to GRPO-based Reinforcement Learning Enhancement
by: Zhao, Pu, et al.
Published: (2024)

Rep2Text: Decoding Full Text from a Single LLM Token Representation
by: Zhao, Haiyan, et al.
Published: (2025)

Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages
by: Li, Zihao, et al.
Published: (2024)

Scaling Teams or Scaling Time? Memory Enabled Lifelong Learning in LLM Multi-Agent Systems
by: Wu, Shanglin, et al.
Published: (2026)

LLM-Safety Evaluations Lack Robustness
by: Beyer, Tim, et al.
Published: (2025)

Bootstrap SGD: Algorithmic Stability and Robustness
by: Christmann, Andreas, et al.
Published: (2024)

Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-Training of Deep Networks
by: Joshi, Siddharth, et al.
Published: (2024)

BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
by: Paglieri, Davide, et al.
Published: (2024)

MDBench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidance
by: Peper, Joseph J., et al.
Published: (2025)

LLM Pretraining Shapes a Generalizable Manifold: Insights into Cross-Modal Transfer to Time Series
by: Roger, Alexis, et al.
Published: (2026)

Prompt Mining for Language-based Human Mobility Forecasting
by: Xue, Hao, et al.
Published: (2024)