Contamination Detection for VLMs using Multi-Modal Semantic Perturbation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911412741210112 |
|---|---|
| author | Park, Jaden Cai, Mu Yao, Feng Shang, Jingbo Lee, Soochahn Lee, Yong Jae |
| author_facet | Park, Jaden Cai, Mu Yao, Feng Shang, Jingbo Lee, Soochahn Lee, Yong Jae |
| contents | Recent advances in Vision-Language Models (VLMs) have achieved state-of-the-art performance on numerous benchmark tasks. However, the use of internet-scale, often proprietary, pretraining corpora raises a critical concern for both practitioners and users: inflated performance due to test-set leakage. While prior works have proposed mitigation strategies such as decontamination of pretraining data and benchmark redesign for LLMs, the complementary direction of developing detection methods for contaminated VLMs remains underexplored. To address this gap, we deliberately contaminate open-source VLMs on popular benchmarks and show that existing detection approaches either fail outright or exhibit inconsistent behavior. We then propose a novel simple yet effective detection method based on multi-modal semantic perturbation, demonstrating that contaminated models fail to generalize under controlled perturbations. Finally, we validate our approach across multiple realistic contamination strategies, confirming its robustness and effectiveness. The code and perturbed dataset are released at https://github.com/jadenpark0/mm-perturb. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_03774 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Contamination Detection for VLMs using Multi-Modal Semantic Perturbation Park, Jaden Cai, Mu Yao, Feng Shang, Jingbo Lee, Soochahn Lee, Yong Jae Machine Learning Recent advances in Vision-Language Models (VLMs) have achieved state-of-the-art performance on numerous benchmark tasks. However, the use of internet-scale, often proprietary, pretraining corpora raises a critical concern for both practitioners and users: inflated performance due to test-set leakage. While prior works have proposed mitigation strategies such as decontamination of pretraining data and benchmark redesign for LLMs, the complementary direction of developing detection methods for contaminated VLMs remains underexplored. To address this gap, we deliberately contaminate open-source VLMs on popular benchmarks and show that existing detection approaches either fail outright or exhibit inconsistent behavior. We then propose a novel simple yet effective detection method based on multi-modal semantic perturbation, demonstrating that contaminated models fail to generalize under controlled perturbations. Finally, we validate our approach across multiple realistic contamination strategies, confirming its robustness and effectiveness. The code and perturbed dataset are released at https://github.com/jadenpark0/mm-perturb. |
| title | Contamination Detection for VLMs using Multi-Modal Semantic Perturbation |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2511.03774 |