Clebsch-Gordan Transformer: Fast and Global Equivariant Attention
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916975037382656 |
|---|---|
| author | Howell, Owen Lewis Zhao, Linfeng Zhu, Xupeng Qian, Yaoyao Huang, Haojie Sun, Lingfeng Thomason, Wil Platt, Robert Walters, Robin |
| author_facet | Howell, Owen Lewis Zhao, Linfeng Zhu, Xupeng Qian, Yaoyao Huang, Haojie Sun, Lingfeng Thomason, Wil Platt, Robert Walters, Robin |
| contents | The global attention mechanism is one of the keys to the success of transformer architecture, but it incurs quadratic computational costs in relation to the number of tokens. On the other hand, equivariant models, which leverage the underlying geometric structures of problem instance, often achieve superior accuracy in physical, biochemical, computer vision, and robotic tasks, at the cost of additional compute requirements. As a result, existing equivariant transformers only support low-order equivariant features and local context windows, limiting their expressiveness and performance. This work proposes Clebsch-Gordan Transformer, achieving efficient global attention by a novel Clebsch-Gordon Convolution on $\SO(3)$ irreducible representations. Our method enables equivariant modeling of features at all orders while achieving ${O}(N \log N)$ input token complexity. Additionally, the proposed method scales well with high-order irreducible features, by exploiting the sparsity of the Clebsch-Gordon matrix. Lastly, we also incorporate optional token permutation equivariance through either weight sharing or data augmentation. We benchmark our method on a diverse set of benchmarks including n-body simulation, QM9, ModelNet point cloud classification and a robotic grasping dataset, showing clear gains over existing equivariant transformers in GPU memory size, speed, and accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_24093 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Clebsch-Gordan Transformer: Fast and Global Equivariant Attention Howell, Owen Lewis Zhao, Linfeng Zhu, Xupeng Qian, Yaoyao Huang, Haojie Sun, Lingfeng Thomason, Wil Platt, Robert Walters, Robin Machine Learning Computer Vision and Pattern Recognition Robotics The global attention mechanism is one of the keys to the success of transformer architecture, but it incurs quadratic computational costs in relation to the number of tokens. On the other hand, equivariant models, which leverage the underlying geometric structures of problem instance, often achieve superior accuracy in physical, biochemical, computer vision, and robotic tasks, at the cost of additional compute requirements. As a result, existing equivariant transformers only support low-order equivariant features and local context windows, limiting their expressiveness and performance. This work proposes Clebsch-Gordan Transformer, achieving efficient global attention by a novel Clebsch-Gordon Convolution on $\SO(3)$ irreducible representations. Our method enables equivariant modeling of features at all orders while achieving ${O}(N \log N)$ input token complexity. Additionally, the proposed method scales well with high-order irreducible features, by exploiting the sparsity of the Clebsch-Gordon matrix. Lastly, we also incorporate optional token permutation equivariance through either weight sharing or data augmentation. We benchmark our method on a diverse set of benchmarks including n-body simulation, QM9, ModelNet point cloud classification and a robotic grasping dataset, showing clear gains over existing equivariant transformers in GPU memory size, speed, and accuracy. |
| title | Clebsch-Gordan Transformer: Fast and Global Equivariant Attention |
| topic | Machine Learning Computer Vision and Pattern Recognition Robotics |
| url | https://arxiv.org/abs/2509.24093 |