Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.10863 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915854249099264 |
|---|---|
| author | Chen, Lin Ni, Bolin Yang, Qi Wang, Zili Ding, Kun Wang, Ying Peng, Houwen Xiang, Shiming |
| author_facet | Chen, Lin Ni, Bolin Yang, Qi Wang, Zili Ding, Kun Wang, Ying Peng, Houwen Xiang, Shiming |
| contents | Despite the remarkable capabilities of Multimodal Large Language Models (MLLMs), they still suffer from visual fading in long-context scenarios. Specifically, the attention to visual tokens diminishes as the text sequence lengthens, leading to text generation detached from visual constraints. We attribute this degradation to the inherent inductive bias of Multimodal RoPE, which penalizes inter-modal attention as the distance between visual and text tokens increases. To address this, we propose inter-modal Distance Invariant Position Encoding (DIPE), a simple but effective mechanism that disentangles position encoding based on modality interactions. DIPE retains the natural relative positioning for intra-modal interactions to preserve local structure, while enforcing an anchored perceptual proximity for inter-modal interactions. This strategy effectively mitigates the inter-modal distance-based penalty, ensuring that visual signals remain perceptually consistent regardless of the context length. Experimental results demonstrate that by integrating DIPE with Multimodal RoPE, the model maintains stable visual grounding in long-context scenarios, significantly alleviating visual fading while preserving performance on standard short-context benchmarks. Code is available at https://github.com/lchen1019/DIPE. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_10863 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Beyond Sequential Distance: Inter-Modal Distance Invariant Position Encoding Chen, Lin Ni, Bolin Yang, Qi Wang, Zili Ding, Kun Wang, Ying Peng, Houwen Xiang, Shiming Computer Vision and Pattern Recognition Despite the remarkable capabilities of Multimodal Large Language Models (MLLMs), they still suffer from visual fading in long-context scenarios. Specifically, the attention to visual tokens diminishes as the text sequence lengthens, leading to text generation detached from visual constraints. We attribute this degradation to the inherent inductive bias of Multimodal RoPE, which penalizes inter-modal attention as the distance between visual and text tokens increases. To address this, we propose inter-modal Distance Invariant Position Encoding (DIPE), a simple but effective mechanism that disentangles position encoding based on modality interactions. DIPE retains the natural relative positioning for intra-modal interactions to preserve local structure, while enforcing an anchored perceptual proximity for inter-modal interactions. This strategy effectively mitigates the inter-modal distance-based penalty, ensuring that visual signals remain perceptually consistent regardless of the context length. Experimental results demonstrate that by integrating DIPE with Multimodal RoPE, the model maintains stable visual grounding in long-context scenarios, significantly alleviating visual fading while preserving performance on standard short-context benchmarks. Code is available at https://github.com/lchen1019/DIPE. |
| title | Beyond Sequential Distance: Inter-Modal Distance Invariant Position Encoding |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2603.10863 |