Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.16840 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914402749382656 |
|---|---|
| author | Pawlowsky, Moritz Vamvakeros, Antonis Weiss, Alexander Bielefeld, Anja Cooper, Samuel J. Docherty, Ronan |
| author_facet | Pawlowsky, Moritz Vamvakeros, Antonis Weiss, Alexander Bielefeld, Anja Cooper, Samuel J. Docherty, Ronan |
| contents | Vision transformers (ViTs) - especially feature foundation models like DINOv2 - learn rich representations useful for many downstream tasks. However, architectural choices (such as positional encoding) can lead to these models displaying positional biases and artefacts independent of semantic content. This makes zero-shot adaption difficult in fields like material science, where images are often cross-sections of homogeneous microstructure (i.e. having no preferred direction). In this work, we investigate the positional bias in ViTs via linear probing, finding it present across a range of objectives and positional encodings, and subsequently reduce it by finetuning models to use ALiBi relative positional encoding. We demonstrate that these models retain desirable general semantics and their unbiased features can be used successfully in trainable segmentation of complex microscopy images. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_16840 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformers Pawlowsky, Moritz Vamvakeros, Antonis Weiss, Alexander Bielefeld, Anja Cooper, Samuel J. Docherty, Ronan Computer Vision and Pattern Recognition Materials Science Vision transformers (ViTs) - especially feature foundation models like DINOv2 - learn rich representations useful for many downstream tasks. However, architectural choices (such as positional encoding) can lead to these models displaying positional biases and artefacts independent of semantic content. This makes zero-shot adaption difficult in fields like material science, where images are often cross-sections of homogeneous microstructure (i.e. having no preferred direction). In this work, we investigate the positional bias in ViTs via linear probing, finding it present across a range of objectives and positional encodings, and subsequently reduce it by finetuning models to use ALiBi relative positional encoding. We demonstrate that these models retain desirable general semantics and their unbiased features can be used successfully in trainable segmentation of complex microscopy images. |
| title | What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformers |
| topic | Computer Vision and Pattern Recognition Materials Science |
| url | https://arxiv.org/abs/2603.16840 |