Masked Attention as a Mechanism for Improving Interpretability of Vision Transformers
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916227462463488 |
|---|---|
| author | Grisi, Clément Litjens, Geert van der Laak, Jeroen |
| author_facet | Grisi, Clément Litjens, Geert van der Laak, Jeroen |
| contents | Vision Transformers are at the heart of the current surge of interest in foundation models for histopathology. They process images by breaking them into smaller patches following a regular grid, regardless of their content. Yet, not all parts of an image are equally relevant for its understanding. This is particularly true in computational pathology where background is completely non-informative and may introduce artefacts that could mislead predictions. To address this issue, we propose a novel method that explicitly masks background in Vision Transformers' attention mechanism. This ensures tokens corresponding to background patches do not contribute to the final image representation, thereby improving model robustness and interpretability. We validate our approach using prostate cancer grading from whole-slide images as a case study. Our results demonstrate that it achieves comparable performance with plain self-attention while providing more accurate and clinically meaningful attention heatmaps. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_18152 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Masked Attention as a Mechanism for Improving Interpretability of Vision Transformers Grisi, Clément Litjens, Geert van der Laak, Jeroen Computer Vision and Pattern Recognition Image and Video Processing 68T07 Vision Transformers are at the heart of the current surge of interest in foundation models for histopathology. They process images by breaking them into smaller patches following a regular grid, regardless of their content. Yet, not all parts of an image are equally relevant for its understanding. This is particularly true in computational pathology where background is completely non-informative and may introduce artefacts that could mislead predictions. To address this issue, we propose a novel method that explicitly masks background in Vision Transformers' attention mechanism. This ensures tokens corresponding to background patches do not contribute to the final image representation, thereby improving model robustness and interpretability. We validate our approach using prostate cancer grading from whole-slide images as a case study. Our results demonstrate that it achieves comparable performance with plain self-attention while providing more accurate and clinically meaningful attention heatmaps. |
| title | Masked Attention as a Mechanism for Improving Interpretability of Vision Transformers |
| topic | Computer Vision and Pattern Recognition Image and Video Processing 68T07 |
| url | https://arxiv.org/abs/2404.18152 |