VideoRoPE: What Makes for Good Video Rotary Position Embedding?
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912403108659200 |
|---|---|
| author | Wei, Xilin Liu, Xiaoran Zang, Yuhang Dong, Xiaoyi Zhang, Pan Cao, Yuhang Tong, Jian Duan, Haodong Guo, Qipeng Wang, Jiaqi Qiu, Xipeng Lin, Dahua |
| author_facet | Wei, Xilin Liu, Xiaoran Zang, Yuhang Dong, Xiaoyi Zhang, Pan Cao, Yuhang Tong, Jian Duan, Haodong Guo, Qipeng Wang, Jiaqi Qiu, Xipeng Lin, Dahua |
| contents | While Rotary Position Embedding (RoPE) and its variants are widely adopted for their long-context capabilities, the extension of the 1D RoPE to video, with its complex spatio-temporal structure, remains an open challenge. This work first introduces a comprehensive analysis that identifies four key characteristics essential for the effective adaptation of RoPE to video, which have not been fully considered in prior work. As part of our analysis, we introduce a challenging V-NIAH-D (Visual Needle-In-A-Haystack with Distractors) task, which adds periodic distractors into V-NIAH. The V-NIAH-D task demonstrates that previous RoPE variants, lacking appropriate temporal dimension allocation, are easily misled by distractors. Based on our analysis, we introduce \textbf{VideoRoPE}, with a \textit{3D structure} designed to preserve spatio-temporal relationships. VideoRoPE features \textit{low-frequency temporal allocation} to mitigate periodic oscillations, a \textit{diagonal layout} to maintain spatial symmetry, and \textit{adjustable temporal spacing} to decouple temporal and spatial indexing. VideoRoPE consistently surpasses previous RoPE variants, across diverse downstream tasks such as long video retrieval, video understanding, and video hallucination. Our code will be available at \href{https://github.com/Wiselnn570/VideoRoPE}{https://github.com/Wiselnn570/VideoRoPE}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_05173 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VideoRoPE: What Makes for Good Video Rotary Position Embedding? Wei, Xilin Liu, Xiaoran Zang, Yuhang Dong, Xiaoyi Zhang, Pan Cao, Yuhang Tong, Jian Duan, Haodong Guo, Qipeng Wang, Jiaqi Qiu, Xipeng Lin, Dahua Computer Vision and Pattern Recognition While Rotary Position Embedding (RoPE) and its variants are widely adopted for their long-context capabilities, the extension of the 1D RoPE to video, with its complex spatio-temporal structure, remains an open challenge. This work first introduces a comprehensive analysis that identifies four key characteristics essential for the effective adaptation of RoPE to video, which have not been fully considered in prior work. As part of our analysis, we introduce a challenging V-NIAH-D (Visual Needle-In-A-Haystack with Distractors) task, which adds periodic distractors into V-NIAH. The V-NIAH-D task demonstrates that previous RoPE variants, lacking appropriate temporal dimension allocation, are easily misled by distractors. Based on our analysis, we introduce \textbf{VideoRoPE}, with a \textit{3D structure} designed to preserve spatio-temporal relationships. VideoRoPE features \textit{low-frequency temporal allocation} to mitigate periodic oscillations, a \textit{diagonal layout} to maintain spatial symmetry, and \textit{adjustable temporal spacing} to decouple temporal and spatial indexing. VideoRoPE consistently surpasses previous RoPE variants, across diverse downstream tasks such as long video retrieval, video understanding, and video hallucination. Our code will be available at \href{https://github.com/Wiselnn570/VideoRoPE}{https://github.com/Wiselnn570/VideoRoPE}. |
| title | VideoRoPE: What Makes for Good Video Rotary Position Embedding? |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2502.05173 |