| _version_ | 1866901211708391424 |
|---|---|
| author | Revista, Zen IA, 10 |
| author_facet | Revista, Zen IA, 10 |
| contents | Transformer models have revolutionized natural language processing and computer vision due to their ability to capture long-range dependencies through the attention mechanism. However, the quadratic computational complexity of the attention mechanism with respect to sequence length poses a significant bottleneck for processing long sequences. This paper introduces a novel approach to reduce the computational complexity of attention to sublinear by leveraging learnable orthogonal projections. Our method projects the query, key, and value matrices into a lower-dimensional subspace using orthogonal projection matrices that are learned during training. By enforcing orthogonality, we ensure that the information captured in the lower-dimensional space is maximally preserved, while the reduced dimensionality leads to a significant reduction in computational cost. We present a detailed theoretical analysis of the proposed method, demonstrating its ability to approximate the original attention mechanism with provable guarantees. Empirical results on various benchmark datasets demonstrate that our approach achieves significant speedups compared to the standard attention mechanism, while maintaining comparable or even improved accuracy. This work opens up new possibilities for scaling transformer models to handle extremely long sequences, paving the way for more efficient and powerful language models. |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_17818930 |
| institution | Zenodo |
| language | |
| publishDate | 2025 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | Beyond Quadratic: Sublinear Attention via Learnable Orthogonal Projections Revista, Zen IA, 10 Transformer models have revolutionized natural language processing and computer vision due to their ability to capture long-range dependencies through the attention mechanism. However, the quadratic computational complexity of the attention mechanism with respect to sequence length poses a significant bottleneck for processing long sequences. This paper introduces a novel approach to reduce the computational complexity of attention to sublinear by leveraging learnable orthogonal projections. Our method projects the query, key, and value matrices into a lower-dimensional subspace using orthogonal projection matrices that are learned during training. By enforcing orthogonality, we ensure that the information captured in the lower-dimensional space is maximally preserved, while the reduced dimensionality leads to a significant reduction in computational cost. We present a detailed theoretical analysis of the proposed method, demonstrating its ability to approximate the original attention mechanism with provable guarantees. Empirical results on various benchmark datasets demonstrate that our approach achieves significant speedups compared to the standard attention mechanism, while maintaining comparable or even improved accuracy. This work opens up new possibilities for scaling transformer models to handle extremely long sequences, paving the way for more efficient and powerful language models. |
| title | Beyond Quadratic: Sublinear Attention via Learnable Orthogonal Projections |
| url | https://doi.org/10.5281/zenodo.17818930 |