PARIC: Probabilistic Attention Regularization for Language Guided Image Classification from Pre-trained Vison Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912274682216448 |
|---|---|
| author | Nautiyal, Mayank Gheorghe, Stela Arranz Stefa, Kristiana Ju, Li Sintorn, Ida-Maria Singh, Prashant |
| author_facet | Nautiyal, Mayank Gheorghe, Stela Arranz Stefa, Kristiana Ju, Li Sintorn, Ida-Maria Singh, Prashant |
| contents | Language-guided attention frameworks have significantly enhanced both interpretability and performance in image classification; however, the reliance on deterministic embeddings from pre-trained vision-language foundation models to generate reference attention maps frequently overlooks the intrinsic multivaluedness and ill-posed characteristics of cross-modal mappings. To address these limitations, we introduce PARIC, a probabilistic framework for guiding visual attention via language specifications. Our approach enables pre-trained vision-language models to generate probabilistic reference attention maps, which align textual and visual modalities more effectively while incorporating uncertainty estimates, as compared to their deterministic counterparts. Experiments on benchmark test problems demonstrate that PARIC enhances prediction accuracy, mitigates bias, ensures consistent predictions, and improves robustness across various datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_11360 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PARIC: Probabilistic Attention Regularization for Language Guided Image Classification from Pre-trained Vison Language Models Nautiyal, Mayank Gheorghe, Stela Arranz Stefa, Kristiana Ju, Li Sintorn, Ida-Maria Singh, Prashant Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Language-guided attention frameworks have significantly enhanced both interpretability and performance in image classification; however, the reliance on deterministic embeddings from pre-trained vision-language foundation models to generate reference attention maps frequently overlooks the intrinsic multivaluedness and ill-posed characteristics of cross-modal mappings. To address these limitations, we introduce PARIC, a probabilistic framework for guiding visual attention via language specifications. Our approach enables pre-trained vision-language models to generate probabilistic reference attention maps, which align textual and visual modalities more effectively while incorporating uncertainty estimates, as compared to their deterministic counterparts. Experiments on benchmark test problems demonstrate that PARIC enhances prediction accuracy, mitigates bias, ensures consistent predictions, and improves robustness across various datasets. |
| title | PARIC: Probabilistic Attention Regularization for Language Guided Image Classification from Pre-trained Vison Language Models |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2503.11360 |