PADRe: A Unifying Polynomial Attention Drop-in Replacement for Efficient Vision Transformer
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914872224120832 |
|---|---|
| author | Letourneau, Pierre-David Singh, Manish Kumar Cheng, Hsin-Pai Han, Shizhong Shi, Yunxiao Jones, Dalton Langston, Matthew Harper Cai, Hong Porikli, Fatih |
| author_facet | Letourneau, Pierre-David Singh, Manish Kumar Cheng, Hsin-Pai Han, Shizhong Shi, Yunxiao Jones, Dalton Langston, Matthew Harper Cai, Hong Porikli, Fatih |
| contents | We present Polynomial Attention Drop-in Replacement (PADRe), a novel and unifying framework designed to replace the conventional self-attention mechanism in transformer models. Notably, several recent alternative attention mechanisms, including Hyena, Mamba, SimA, Conv2Former, and Castling-ViT, can be viewed as specific instances of our PADRe framework. PADRe leverages polynomial functions and draws upon established results from approximation theory, enhancing computational efficiency without compromising accuracy. PADRe's key components include multiplicative nonlinearities, which we implement using straightforward, hardware-friendly operations such as Hadamard products, incurring only linear computational and memory costs. PADRe further avoids the need for using complex functions such as Softmax, yet it maintains comparable or superior accuracy compared to traditional self-attention. We assess the effectiveness of PADRe as a drop-in replacement for self-attention across diverse computer vision tasks. These tasks include image classification, image-based 2D object detection, and 3D point cloud object detection. Empirical results demonstrate that PADRe runs significantly faster than the conventional self-attention (11x ~ 43x faster on server GPU and mobile NPU) while maintaining similar accuracy when substituting self-attention in the transformer models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_11306 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | PADRe: A Unifying Polynomial Attention Drop-in Replacement for Efficient Vision Transformer Letourneau, Pierre-David Singh, Manish Kumar Cheng, Hsin-Pai Han, Shizhong Shi, Yunxiao Jones, Dalton Langston, Matthew Harper Cai, Hong Porikli, Fatih Computer Vision and Pattern Recognition We present Polynomial Attention Drop-in Replacement (PADRe), a novel and unifying framework designed to replace the conventional self-attention mechanism in transformer models. Notably, several recent alternative attention mechanisms, including Hyena, Mamba, SimA, Conv2Former, and Castling-ViT, can be viewed as specific instances of our PADRe framework. PADRe leverages polynomial functions and draws upon established results from approximation theory, enhancing computational efficiency without compromising accuracy. PADRe's key components include multiplicative nonlinearities, which we implement using straightforward, hardware-friendly operations such as Hadamard products, incurring only linear computational and memory costs. PADRe further avoids the need for using complex functions such as Softmax, yet it maintains comparable or superior accuracy compared to traditional self-attention. We assess the effectiveness of PADRe as a drop-in replacement for self-attention across diverse computer vision tasks. These tasks include image classification, image-based 2D object detection, and 3D point cloud object detection. Empirical results demonstrate that PADRe runs significantly faster than the conventional self-attention (11x ~ 43x faster on server GPU and mobile NPU) while maintaining similar accuracy when substituting self-attention in the transformer models. |
| title | PADRe: A Unifying Polynomial Attention Drop-in Replacement for Efficient Vision Transformer |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.11306 |