Enhancing Concept Localization in CLIP-based Concept Bottleneck Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918156673482752 |
|---|---|
| author | Kazmierczak, Rémi Azzolin, Steve Berthier, Eloïse Frehse, Goran Franchi, Gianni |
| author_facet | Kazmierczak, Rémi Azzolin, Steve Berthier, Eloïse Frehse, Goran Franchi, Gianni |
| contents | This paper addresses explainable AI (XAI) through the lens of Concept Bottleneck Models (CBMs) that do not require explicit concept annotations, relying instead on concepts extracted using CLIP in a zero-shot manner. We show that CLIP, which is central in these techniques, is prone to concept hallucination, incorrectly predicting the presence or absence of concepts within an image in scenarios used in numerous CBMs, hence undermining the faithfulness of explanations. To mitigate this issue, we introduce Concept Hallucination Inhibition via Localized Interpretability (CHILI), a technique that disentangles image embeddings and localizes pixels corresponding to target concepts. Furthermore, our approach supports the generation of saliency-based explanations that are more interpretable. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_07115 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Enhancing Concept Localization in CLIP-based Concept Bottleneck Models Kazmierczak, Rémi Azzolin, Steve Berthier, Eloïse Frehse, Goran Franchi, Gianni Computer Vision and Pattern Recognition This paper addresses explainable AI (XAI) through the lens of Concept Bottleneck Models (CBMs) that do not require explicit concept annotations, relying instead on concepts extracted using CLIP in a zero-shot manner. We show that CLIP, which is central in these techniques, is prone to concept hallucination, incorrectly predicting the presence or absence of concepts within an image in scenarios used in numerous CBMs, hence undermining the faithfulness of explanations. To mitigate this issue, we introduce Concept Hallucination Inhibition via Localized Interpretability (CHILI), a technique that disentangles image embeddings and localizes pixels corresponding to target concepts. Furthermore, our approach supports the generation of saliency-based explanations that are more interpretable. |
| title | Enhancing Concept Localization in CLIP-based Concept Bottleneck Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2510.07115 |