Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912221218471936 |
|---|---|
| author | Schenck, Connor Reid, Isaac Jacob, Mithun George Bewley, Alex Ainslie, Joshua Rendleman, David Jain, Deepali Sharma, Mohit Dubey, Avinava Wahid, Ayzaan Singh, Sumeet Wagner, René Ding, Tianli Fu, Chuyuan Byravan, Arunkumar Varley, Jake Gritsenko, Alexey Minderer, Matthias Kalashnikov, Dmitry Tompson, Jonathan Sindhwani, Vikas Choromanski, Krzysztof |
| author_facet | Schenck, Connor Reid, Isaac Jacob, Mithun George Bewley, Alex Ainslie, Joshua Rendleman, David Jain, Deepali Sharma, Mohit Dubey, Avinava Wahid, Ayzaan Singh, Sumeet Wagner, René Ding, Tianli Fu, Chuyuan Byravan, Arunkumar Varley, Jake Gritsenko, Alexey Minderer, Matthias Kalashnikov, Dmitry Tompson, Jonathan Sindhwani, Vikas Choromanski, Krzysztof |
| contents | We introduce STRING: Separable Translationally Invariant Position Encodings. STRING extends Rotary Position Encodings, a recently proposed and widely used algorithm in large language models, via a unifying theoretical framework. Importantly, STRING still provides exact translation invariance, including token coordinates of arbitrary dimensionality, whilst maintaining a low computational footprint. These properties are especially important in robotics, where efficient 3D token representation is key. We integrate STRING into Vision Transformers with RGB(-D) inputs (color plus optional depth), showing substantial gains, e.g. in open-vocabulary object detection and for robotics controllers. We complement our experiments with a rigorous mathematical analysis, proving the universality of our methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_02562 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Learning the RoPEs: Better 2D and 3D Position Encodings with STRING Schenck, Connor Reid, Isaac Jacob, Mithun George Bewley, Alex Ainslie, Joshua Rendleman, David Jain, Deepali Sharma, Mohit Dubey, Avinava Wahid, Ayzaan Singh, Sumeet Wagner, René Ding, Tianli Fu, Chuyuan Byravan, Arunkumar Varley, Jake Gritsenko, Alexey Minderer, Matthias Kalashnikov, Dmitry Tompson, Jonathan Sindhwani, Vikas Choromanski, Krzysztof Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Robotics We introduce STRING: Separable Translationally Invariant Position Encodings. STRING extends Rotary Position Encodings, a recently proposed and widely used algorithm in large language models, via a unifying theoretical framework. Importantly, STRING still provides exact translation invariance, including token coordinates of arbitrary dimensionality, whilst maintaining a low computational footprint. These properties are especially important in robotics, where efficient 3D token representation is key. We integrate STRING into Vision Transformers with RGB(-D) inputs (color plus optional depth), showing substantial gains, e.g. in open-vocabulary object detection and for robotics controllers. We complement our experiments with a rigorous mathematical analysis, proving the universality of our methods. |
| title | Learning the RoPEs: Better 2D and 3D Position Encodings with STRING |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Robotics |
| url | https://arxiv.org/abs/2502.02562 |