DeBiFormer: Vision Transformer with Deformable Agent Bi-level Routing Attention
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910645310455808 |
|---|---|
| author | Long, Nguyen Huu Bao Zhang, Chenyu Shi, Yuzhi Hirakawa, Tsubasa Yamashita, Takayoshi Matsui, Tohgoroh Fujiyoshi, Hironobu |
| author_facet | Long, Nguyen Huu Bao Zhang, Chenyu Shi, Yuzhi Hirakawa, Tsubasa Yamashita, Takayoshi Matsui, Tohgoroh Fujiyoshi, Hironobu |
| contents | Vision Transformers with various attention modules have demonstrated superior performance on vision tasks. While using sparsity-adaptive attention, such as in DAT, has yielded strong results in image classification, the key-value pairs selected by deformable points lack semantic relevance when fine-tuning for semantic segmentation tasks. The query-aware sparsity attention in BiFormer seeks to focus each query on top-k routed regions. However, during attention calculation, the selected key-value pairs are influenced by too many irrelevant queries, reducing attention on the more important ones. To address these issues, we propose the Deformable Bi-level Routing Attention (DBRA) module, which optimizes the selection of key-value pairs using agent queries and enhances the interpretability of queries in attention maps. Based on this, we introduce the Deformable Bi-level Routing Attention Transformer (DeBiFormer), a novel general-purpose vision transformer built with the DBRA module. DeBiFormer has been validated on various computer vision tasks, including image classification, object detection, and semantic segmentation, providing strong evidence of its effectiveness.Code is available at {https://github.com/maclong01/DeBiFormer} |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_08582 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DeBiFormer: Vision Transformer with Deformable Agent Bi-level Routing Attention Long, Nguyen Huu Bao Zhang, Chenyu Shi, Yuzhi Hirakawa, Tsubasa Yamashita, Takayoshi Matsui, Tohgoroh Fujiyoshi, Hironobu Computer Vision and Pattern Recognition Vision Transformers with various attention modules have demonstrated superior performance on vision tasks. While using sparsity-adaptive attention, such as in DAT, has yielded strong results in image classification, the key-value pairs selected by deformable points lack semantic relevance when fine-tuning for semantic segmentation tasks. The query-aware sparsity attention in BiFormer seeks to focus each query on top-k routed regions. However, during attention calculation, the selected key-value pairs are influenced by too many irrelevant queries, reducing attention on the more important ones. To address these issues, we propose the Deformable Bi-level Routing Attention (DBRA) module, which optimizes the selection of key-value pairs using agent queries and enhances the interpretability of queries in attention maps. Based on this, we introduce the Deformable Bi-level Routing Attention Transformer (DeBiFormer), a novel general-purpose vision transformer built with the DBRA module. DeBiFormer has been validated on various computer vision tasks, including image classification, object detection, and semantic segmentation, providing strong evidence of its effectiveness.Code is available at {https://github.com/maclong01/DeBiFormer} |
| title | DeBiFormer: Vision Transformer with Deformable Agent Bi-level Routing Attention |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.08582 |