Tell me why: Visual foundation models as self-explainable classifiers
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915175484882944 |
|---|---|
| author | Turbé, Hugues Bjelogrlic, Mina Mengaldo, Gianmarco Lovis, Christian |
| author_facet | Turbé, Hugues Bjelogrlic, Mina Mengaldo, Gianmarco Lovis, Christian |
| contents | Visual foundation models (VFMs) have become increasingly popular due to their state-of-the-art performance. However, interpretability remains crucial for critical applications. In this sense, self-explainable models (SEM) aim to provide interpretable classifiers that decompose predictions into a weighted sum of interpretable concepts. Despite their promise, recent studies have shown that these explanations often lack faithfulness. In this work, we combine VFMs with a novel prototypical architecture and specialized training objectives. By training only a lightweight head (approximately 1M parameters) on top of frozen VFMs, our approach (ProtoFM) offers an efficient and interpretable solution. Evaluations demonstrate that our approach achieves competitive classification performance while outperforming existing models across a range of interpretability metrics derived from the literature. Code is available at https://github.com/hturbe/proto-fm. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_19577 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Tell me why: Visual foundation models as self-explainable classifiers Turbé, Hugues Bjelogrlic, Mina Mengaldo, Gianmarco Lovis, Christian Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning 68T10 I.2.6; I.2.10; I.5.4 Visual foundation models (VFMs) have become increasingly popular due to their state-of-the-art performance. However, interpretability remains crucial for critical applications. In this sense, self-explainable models (SEM) aim to provide interpretable classifiers that decompose predictions into a weighted sum of interpretable concepts. Despite their promise, recent studies have shown that these explanations often lack faithfulness. In this work, we combine VFMs with a novel prototypical architecture and specialized training objectives. By training only a lightweight head (approximately 1M parameters) on top of frozen VFMs, our approach (ProtoFM) offers an efficient and interpretable solution. Evaluations demonstrate that our approach achieves competitive classification performance while outperforming existing models across a range of interpretability metrics derived from the literature. Code is available at https://github.com/hturbe/proto-fm. |
| title | Tell me why: Visual foundation models as self-explainable classifiers |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning 68T10 I.2.6; I.2.10; I.5.4 |
| url | https://arxiv.org/abs/2502.19577 |