Geometry without Position? When Positional Embeddings Help and Hurt Spatial Reasoning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910005338308608 |
|---|---|
| author | Shi, Jian Birsak, Michael Cui, Wenqing Li, Zhenyu Wonka, Peter |
| author_facet | Shi, Jian Birsak, Michael Cui, Wenqing Li, Zhenyu Wonka, Peter |
| contents | This paper revisits the role of positional embeddings (PEs) within vision transformers (ViTs) from a geometric perspective. We show that PEs are not mere token indices but effectively function as geometric priors that shape the spatial structure of the representation. We introduce token-level diagnostics that measure how multi-view geometric consistency in ViT representation depends on consitent PEs. Through extensive experiments on 14 foundation ViT models, we reveal how PEs influence multi-view geometry and spatial reasoning. Our findings clarify the role of PEs as a causal mechanism that governs spatial structure in ViT representations. Our code is provided in https://github.com/shijianjian/vit-geometry-probes |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_22231 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Geometry without Position? When Positional Embeddings Help and Hurt Spatial Reasoning Shi, Jian Birsak, Michael Cui, Wenqing Li, Zhenyu Wonka, Peter Computer Vision and Pattern Recognition This paper revisits the role of positional embeddings (PEs) within vision transformers (ViTs) from a geometric perspective. We show that PEs are not mere token indices but effectively function as geometric priors that shape the spatial structure of the representation. We introduce token-level diagnostics that measure how multi-view geometric consistency in ViT representation depends on consitent PEs. Through extensive experiments on 14 foundation ViT models, we reveal how PEs influence multi-view geometry and spatial reasoning. Our findings clarify the role of PEs as a causal mechanism that governs spatial structure in ViT representations. Our code is provided in https://github.com/shijianjian/vit-geometry-probes |
| title | Geometry without Position? When Positional Embeddings Help and Hurt Spatial Reasoning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2601.22231 |