Prompt-CAM: Making Vision Transformers Interpretable for Fine-Grained Analysis
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917979475673088 |
|---|---|
| author | Chowdhury, Arpita Paul, Dipanjyoti Mai, Zheda Gu, Jianyang Zhang, Ziheng Mehrab, Kazi Sajeed Campolongo, Elizabeth G. Rubenstein, Daniel Stewart, Charles V. Karpatne, Anuj Berger-Wolf, Tanya Su, Yu Chao, Wei-Lun |
| author_facet | Chowdhury, Arpita Paul, Dipanjyoti Mai, Zheda Gu, Jianyang Zhang, Ziheng Mehrab, Kazi Sajeed Campolongo, Elizabeth G. Rubenstein, Daniel Stewart, Charles V. Karpatne, Anuj Berger-Wolf, Tanya Su, Yu Chao, Wei-Lun |
| contents | We present a simple approach to make pre-trained Vision Transformers (ViTs) interpretable for fine-grained analysis, aiming to identify and localize the traits that distinguish visually similar categories, such as bird species. Pre-trained ViTs, such as DINO, have demonstrated remarkable capabilities in extracting localized, discriminative features. However, saliency maps like Grad-CAM often fail to identify these traits, producing blurred, coarse heatmaps that highlight entire objects instead. We propose a novel approach, Prompt Class Attention Map (Prompt-CAM), to address this limitation. Prompt-CAM learns class-specific prompts for a pre-trained ViT and uses the corresponding outputs for classification. To correctly classify an image, the true-class prompt must attend to unique image patches not present in other classes' images (i.e., traits). As a result, the true class's multi-head attention maps reveal traits and their locations. Implementation-wise, Prompt-CAM is almost a ``free lunch,'' requiring only a modification to the prediction head of Visual Prompt Tuning (VPT). This makes Prompt-CAM easy to train and apply, in stark contrast to other interpretable methods that require designing specific models and training processes. Extensive empirical studies on a dozen datasets from various domains (e.g., birds, fishes, insects, fungi, flowers, food, and cars) validate the superior interpretation capability of Prompt-CAM. The source code and demo are available at https://github.com/Imageomics/Prompt_CAM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_09333 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Prompt-CAM: Making Vision Transformers Interpretable for Fine-Grained Analysis Chowdhury, Arpita Paul, Dipanjyoti Mai, Zheda Gu, Jianyang Zhang, Ziheng Mehrab, Kazi Sajeed Campolongo, Elizabeth G. Rubenstein, Daniel Stewart, Charles V. Karpatne, Anuj Berger-Wolf, Tanya Su, Yu Chao, Wei-Lun Computer Vision and Pattern Recognition Artificial Intelligence We present a simple approach to make pre-trained Vision Transformers (ViTs) interpretable for fine-grained analysis, aiming to identify and localize the traits that distinguish visually similar categories, such as bird species. Pre-trained ViTs, such as DINO, have demonstrated remarkable capabilities in extracting localized, discriminative features. However, saliency maps like Grad-CAM often fail to identify these traits, producing blurred, coarse heatmaps that highlight entire objects instead. We propose a novel approach, Prompt Class Attention Map (Prompt-CAM), to address this limitation. Prompt-CAM learns class-specific prompts for a pre-trained ViT and uses the corresponding outputs for classification. To correctly classify an image, the true-class prompt must attend to unique image patches not present in other classes' images (i.e., traits). As a result, the true class's multi-head attention maps reveal traits and their locations. Implementation-wise, Prompt-CAM is almost a ``free lunch,'' requiring only a modification to the prediction head of Visual Prompt Tuning (VPT). This makes Prompt-CAM easy to train and apply, in stark contrast to other interpretable methods that require designing specific models and training processes. Extensive empirical studies on a dozen datasets from various domains (e.g., birds, fishes, insects, fungi, flowers, food, and cars) validate the superior interpretation capability of Prompt-CAM. The source code and demo are available at https://github.com/Imageomics/Prompt_CAM. |
| title | Prompt-CAM: Making Vision Transformers Interpretable for Fine-Grained Analysis |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2501.09333 |