MVTamperBench: Evaluating Robustness of Vision-Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909645508968448 |
|---|---|
| author | Agarwal, Amit Panda, Srikant Charles, Angeline Kumar, Bhargava Patel, Hitesh Pattnayak, Priyaranjan Rafi, Taki Hasan Kumar, Tejaswini Meghwani, Hansa Gupta, Karan Chae, Dong-Kyu |
| author_facet | Agarwal, Amit Panda, Srikant Charles, Angeline Kumar, Bhargava Patel, Hitesh Pattnayak, Priyaranjan Rafi, Taki Hasan Kumar, Tejaswini Meghwani, Hansa Gupta, Karan Chae, Dong-Kyu |
| contents | Multimodal Large Language Models (MLLMs), are recent advancement of Vision-Language Models (VLMs) that have driven major advances in video understanding. However, their vulnerability to adversarial tampering and manipulations remains underexplored. To address this gap, we introduce \textbf{MVTamperBench}, a benchmark that systematically evaluates MLLM robustness against five prevalent tampering techniques: rotation, masking, substitution, repetition, and dropping; based on real-world visual tampering scenarios such as surveillance interference, social media content edits, and misinformation injection. MVTamperBench comprises ~3.4K original videos, expanded into over ~17K tampered clips covering 19 distinct video manipulation tasks. This benchmark challenges models to detect manipulations in spatial and temporal coherence. We evaluate 45 recent MLLMs from 15+ model families. We reveal substantial variability in resilience across tampering types and show that larger parameter counts do not necessarily guarantee robustness. MVTamperBench sets a new benchmark for developing tamper-resilient MLLM in safety-critical applications, including detecting clickbait, preventing harmful content distribution, and enforcing policies on media platforms. We release all code, data, and benchmark to foster open research in trustworthy video understanding.
Code: https://amitbcp.github.io/MVTamperBench/ Data: https://huggingface.co/datasets/Srikant86/MVTamperBench |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_19794 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MVTamperBench: Evaluating Robustness of Vision-Language Models Agarwal, Amit Panda, Srikant Charles, Angeline Kumar, Bhargava Patel, Hitesh Pattnayak, Priyaranjan Rafi, Taki Hasan Kumar, Tejaswini Meghwani, Hansa Gupta, Karan Chae, Dong-Kyu Computer Vision and Pattern Recognition 68T37, 68T05, 68Q32, 68T45, 94A08, 68T40, 68Q85 I.2.10; I.2.7; I.5.4; I.4.9; I.4.8; H.5.1 Multimodal Large Language Models (MLLMs), are recent advancement of Vision-Language Models (VLMs) that have driven major advances in video understanding. However, their vulnerability to adversarial tampering and manipulations remains underexplored. To address this gap, we introduce \textbf{MVTamperBench}, a benchmark that systematically evaluates MLLM robustness against five prevalent tampering techniques: rotation, masking, substitution, repetition, and dropping; based on real-world visual tampering scenarios such as surveillance interference, social media content edits, and misinformation injection. MVTamperBench comprises ~3.4K original videos, expanded into over ~17K tampered clips covering 19 distinct video manipulation tasks. This benchmark challenges models to detect manipulations in spatial and temporal coherence. We evaluate 45 recent MLLMs from 15+ model families. We reveal substantial variability in resilience across tampering types and show that larger parameter counts do not necessarily guarantee robustness. MVTamperBench sets a new benchmark for developing tamper-resilient MLLM in safety-critical applications, including detecting clickbait, preventing harmful content distribution, and enforcing policies on media platforms. We release all code, data, and benchmark to foster open research in trustworthy video understanding. Code: https://amitbcp.github.io/MVTamperBench/ Data: https://huggingface.co/datasets/Srikant86/MVTamperBench |
| title | MVTamperBench: Evaluating Robustness of Vision-Language Models |
| topic | Computer Vision and Pattern Recognition 68T37, 68T05, 68Q32, 68T45, 94A08, 68T40, 68Q85 I.2.10; I.2.7; I.5.4; I.4.9; I.4.8; H.5.1 |
| url | https://arxiv.org/abs/2412.19794 |