PCRI: Measuring Context Robustness in Multimodal Models for Enterprise Applications
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914062136246272 |
|---|---|
| author | Patel, Hitesh Laxmichand Agarwal, Amit Panda, Srikant Meghwani, Hansa Dua, Karan Li, Paul Sheng, Tao Ravi, Sujith Roth, Dan |
| author_facet | Patel, Hitesh Laxmichand Agarwal, Amit Panda, Srikant Meghwani, Hansa Dua, Karan Li, Paul Sheng, Tao Ravi, Sujith Roth, Dan |
| contents | The reliability of Multimodal Large Language Models (MLLMs) in real-world settings is often undermined by sensitivity to irrelevant or distracting visual context, an aspect not captured by existing evaluation metrics. We introduce the \textbf{Patch Context Robustness Index (PCRI)}, the first systematic and interpretable score for quantifying MLLM robustness to variations in visual context granularity, measuring performance changes between localized image patches and full-image input.
Applying PCRI to 19 state-of-the-art MLLMs across 15 vision-language benchmarks, we find that most leading models remain brittle to background noise, with only a few, such as InternVL2-26B and Qwen2VL-72B, demonstrating consistent robustness across tasks. PCRI analysis also highlights how different model architectures handle and integrate visual context, offering actionable diagnostic insight for both researchers and practitioners.
PCRI enables rigorous comparison of context robustness, supporting principled model selection and guiding the development of future architectures and training strategies for robust, real-world deployment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_23879 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PCRI: Measuring Context Robustness in Multimodal Models for Enterprise Applications Patel, Hitesh Laxmichand Agarwal, Amit Panda, Srikant Meghwani, Hansa Dua, Karan Li, Paul Sheng, Tao Ravi, Sujith Roth, Dan Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Multimedia 68T50, 68T45 I.2.7; I.2.10; I.4.8; I.4.10; I.4.0 The reliability of Multimodal Large Language Models (MLLMs) in real-world settings is often undermined by sensitivity to irrelevant or distracting visual context, an aspect not captured by existing evaluation metrics. We introduce the \textbf{Patch Context Robustness Index (PCRI)}, the first systematic and interpretable score for quantifying MLLM robustness to variations in visual context granularity, measuring performance changes between localized image patches and full-image input. Applying PCRI to 19 state-of-the-art MLLMs across 15 vision-language benchmarks, we find that most leading models remain brittle to background noise, with only a few, such as InternVL2-26B and Qwen2VL-72B, demonstrating consistent robustness across tasks. PCRI analysis also highlights how different model architectures handle and integrate visual context, offering actionable diagnostic insight for both researchers and practitioners. PCRI enables rigorous comparison of context robustness, supporting principled model selection and guiding the development of future architectures and training strategies for robust, real-world deployment. |
| title | PCRI: Measuring Context Robustness in Multimodal Models for Enterprise Applications |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Multimedia 68T50, 68T45 I.2.7; I.2.10; I.4.8; I.4.10; I.4.0 |
| url | https://arxiv.org/abs/2509.23879 |