Demystify Transformers & Convolutions in Modern Image Deep Networks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2022
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911012545888256 |
|---|---|
| author | Hu, Xiaowei Shi, Min Wang, Weiyun Wu, Sitong Xing, Linjie Wang, Wenhai Zhu, Xizhou Lu, Lewei Zhou, Jie Wang, Xiaogang Qiao, Yu Dai, Jifeng |
| author_facet | Hu, Xiaowei Shi, Min Wang, Weiyun Wu, Sitong Xing, Linjie Wang, Wenhai Zhu, Xizhou Lu, Lewei Zhou, Jie Wang, Xiaogang Qiao, Yu Dai, Jifeng |
| contents | Vision transformers have gained popularity recently, leading to the development of new vision backbones with improved features and consistent performance gains. However, these advancements are not solely attributable to novel feature transformation designs; certain benefits also arise from advanced network-level and block-level architectures. This paper aims to identify the real gains of popular convolution and attention operators through a detailed study. We find that the key difference among these feature transformation modules, such as attention or convolution, lies in their spatial feature aggregation approach, known as the "spatial token mixer" (STM). To facilitate an impartial comparison, we introduce a unified architecture to neutralize the impact of divergent network-level and block-level designs. Subsequently, various STMs are integrated into this unified framework for comprehensive comparative analysis. Our experiments on various tasks and an analysis of inductive bias show a significant performance boost due to advanced network-level and block-level designs, but performance differences persist among different STMs. Our detailed analysis also reveals various findings about different STMs, including effective receptive fields, invariance, and adversarial robustness tests. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2211_05781 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Demystify Transformers & Convolutions in Modern Image Deep Networks Hu, Xiaowei Shi, Min Wang, Weiyun Wu, Sitong Xing, Linjie Wang, Wenhai Zhu, Xizhou Lu, Lewei Zhou, Jie Wang, Xiaogang Qiao, Yu Dai, Jifeng Computer Vision and Pattern Recognition Vision transformers have gained popularity recently, leading to the development of new vision backbones with improved features and consistent performance gains. However, these advancements are not solely attributable to novel feature transformation designs; certain benefits also arise from advanced network-level and block-level architectures. This paper aims to identify the real gains of popular convolution and attention operators through a detailed study. We find that the key difference among these feature transformation modules, such as attention or convolution, lies in their spatial feature aggregation approach, known as the "spatial token mixer" (STM). To facilitate an impartial comparison, we introduce a unified architecture to neutralize the impact of divergent network-level and block-level designs. Subsequently, various STMs are integrated into this unified framework for comprehensive comparative analysis. Our experiments on various tasks and an analysis of inductive bias show a significant performance boost due to advanced network-level and block-level designs, but performance differences persist among different STMs. Our detailed analysis also reveals various findings about different STMs, including effective receptive fields, invariance, and adversarial robustness tests. |
| title | Demystify Transformers & Convolutions in Modern Image Deep Networks |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2211.05781 |