Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.25803 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910076880551936 |
|---|---|
| author | Baxevanakis, Spiros Karageorgis, Platon Dravilas, Ioannis Szewczyk, Konrad |
| author_facet | Baxevanakis, Spiros Karageorgis, Platon Dravilas, Ioannis Szewczyk, Konrad |
| contents | Training Vision Transformers (ViTs) presents significant challenges, one of which is the emergence of artifacts in attention maps, hindering their interpretability. Darcet et al. (2024) investigated this phenomenon and attributed it to the need of ViTs to store global information beyond the [CLS] token. They proposed a novel solution involving the addition of empty input tokens, named registers, which successfully eliminate artifacts and improve the clarity of attention maps. In this work, we reproduce the findings of Darcet et al. (2024) and evaluate the generalizability of their claims across multiple models, including DINO, DINOv2, OpenCLIP, and DeiT3. While we confirm the validity of several of their key claims, our results reveal that some claims do not extend universally to other models. Additionally, we explore the impact of model size, extending their findings to smaller models. Finally, we untie terminology inconsistencies found in the original paper and explain their impact when generalizing to a wider range of models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_25803 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Do All Vision Transformers Need Registers? A Cross-Architectural Reassessment Baxevanakis, Spiros Karageorgis, Platon Dravilas, Ioannis Szewczyk, Konrad Computer Vision and Pattern Recognition Machine Learning I.2.10; I.4.8; I.5.4 Training Vision Transformers (ViTs) presents significant challenges, one of which is the emergence of artifacts in attention maps, hindering their interpretability. Darcet et al. (2024) investigated this phenomenon and attributed it to the need of ViTs to store global information beyond the [CLS] token. They proposed a novel solution involving the addition of empty input tokens, named registers, which successfully eliminate artifacts and improve the clarity of attention maps. In this work, we reproduce the findings of Darcet et al. (2024) and evaluate the generalizability of their claims across multiple models, including DINO, DINOv2, OpenCLIP, and DeiT3. While we confirm the validity of several of their key claims, our results reveal that some claims do not extend universally to other models. Additionally, we explore the impact of model size, extending their findings to smaller models. Finally, we untie terminology inconsistencies found in the original paper and explain their impact when generalizing to a wider range of models. |
| title | Do All Vision Transformers Need Registers? A Cross-Architectural Reassessment |
| topic | Computer Vision and Pattern Recognition Machine Learning I.2.10; I.4.8; I.5.4 |
| url | https://arxiv.org/abs/2603.25803 |