A 2D Semantic-Aware Position Encoding for Vision Transformers
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916737071448064 |
|---|---|
| author | Chen, Xi Zhou, Shiyang Huang, Muqi Feng, Jiaxu Xiong, Yun Zhou, Kun Yang, Biao Zhang, Yuhui Bao, Huishuai Peng, Sijia Li, Chuan Shi, Feng |
| author_facet | Chen, Xi Zhou, Shiyang Huang, Muqi Feng, Jiaxu Xiong, Yun Zhou, Kun Yang, Biao Zhang, Yuhui Bao, Huishuai Peng, Sijia Li, Chuan Shi, Feng |
| contents | Vision transformers have demonstrated significant advantages in computer vision tasks due to their ability to capture long-range dependencies and contextual relationships through self-attention. However, existing position encoding techniques, which are largely borrowed from natural language processing, fail to effectively capture semantic-aware positional relationships between image patches. Traditional approaches like absolute position encoding and relative position encoding primarily focus on 1D linear position relationship, often neglecting the semantic similarity between distant yet contextually related patches. These limitations hinder model generalization, translation equivariance, and the ability to effectively handle repetitive or structured patterns in images. In this paper, we propose 2-Dimensional Semantic-Aware Position Encoding ($\text{SaPE}^2$), a novel position encoding method with semantic awareness that dynamically adapts position representations by leveraging local content instead of fixed linear position relationship or spatial coordinates. Our method enhances the model's ability to generalize across varying image resolutions and scales, improves translation equivariance, and better aggregates features for visually similar but spatially distant patches. By integrating $\text{SaPE}^2$ into vision transformers, we bridge the gap between position encoding and perceptual similarity, thereby improving performance on computer vision tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_09466 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A 2D Semantic-Aware Position Encoding for Vision Transformers Chen, Xi Zhou, Shiyang Huang, Muqi Feng, Jiaxu Xiong, Yun Zhou, Kun Yang, Biao Zhang, Yuhui Bao, Huishuai Peng, Sijia Li, Chuan Shi, Feng Computer Vision and Pattern Recognition Artificial Intelligence Vision transformers have demonstrated significant advantages in computer vision tasks due to their ability to capture long-range dependencies and contextual relationships through self-attention. However, existing position encoding techniques, which are largely borrowed from natural language processing, fail to effectively capture semantic-aware positional relationships between image patches. Traditional approaches like absolute position encoding and relative position encoding primarily focus on 1D linear position relationship, often neglecting the semantic similarity between distant yet contextually related patches. These limitations hinder model generalization, translation equivariance, and the ability to effectively handle repetitive or structured patterns in images. In this paper, we propose 2-Dimensional Semantic-Aware Position Encoding ($\text{SaPE}^2$), a novel position encoding method with semantic awareness that dynamically adapts position representations by leveraging local content instead of fixed linear position relationship or spatial coordinates. Our method enhances the model's ability to generalize across varying image resolutions and scales, improves translation equivariance, and better aggregates features for visually similar but spatially distant patches. By integrating $\text{SaPE}^2$ into vision transformers, we bridge the gap between position encoding and perceptual similarity, thereby improving performance on computer vision tasks. |
| title | A 2D Semantic-Aware Position Encoding for Vision Transformers |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2505.09466 |