Why mamba is effective? Exploit Linear Transformer-Mamba Network for Multi-Modality Image Fusion
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910590453153792 |
|---|---|
| author | Zhu, Chenguang Gao, Shan Chen, Huafeng Guo, Guangqian Wang, Chaowei Wang, Yaoxing Lei, Chen Shu Fan, Quanjiang |
| author_facet | Zhu, Chenguang Gao, Shan Chen, Huafeng Guo, Guangqian Wang, Chaowei Wang, Yaoxing Lei, Chen Shu Fan, Quanjiang |
| contents | Multi-modality image fusion aims to integrate the merits of images from different sources and render high-quality fusion images. However, existing feature extraction and fusion methods are either constrained by inherent local reduction bias and static parameters during inference (CNN) or limited by quadratic computational complexity (Transformers), and cannot effectively extract and fuse features. To solve this problem, we propose a dual-branch image fusion network called Tmamba. It consists of linear Transformer and Mamba, which has global modeling capabilities while maintaining linear complexity. Due to the difference between the Transformer and Mamba structures, the features extracted by the two branches carry channel and position information respectively. T-M interaction structure is designed between the two branches, using global learnable parameters and convolutional layers to transfer position and channel information respectively. We further propose cross-modal interaction at the attention level to obtain cross-modal attention. Experiments show that our Tmamba achieves promising results in multiple fusion tasks, including infrared-visible image fusion and medical image fusion. Code with checkpoints will be available after the peer-review process. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_03223 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Why mamba is effective? Exploit Linear Transformer-Mamba Network for Multi-Modality Image Fusion Zhu, Chenguang Gao, Shan Chen, Huafeng Guo, Guangqian Wang, Chaowei Wang, Yaoxing Lei, Chen Shu Fan, Quanjiang Computer Vision and Pattern Recognition Multi-modality image fusion aims to integrate the merits of images from different sources and render high-quality fusion images. However, existing feature extraction and fusion methods are either constrained by inherent local reduction bias and static parameters during inference (CNN) or limited by quadratic computational complexity (Transformers), and cannot effectively extract and fuse features. To solve this problem, we propose a dual-branch image fusion network called Tmamba. It consists of linear Transformer and Mamba, which has global modeling capabilities while maintaining linear complexity. Due to the difference between the Transformer and Mamba structures, the features extracted by the two branches carry channel and position information respectively. T-M interaction structure is designed between the two branches, using global learnable parameters and convolutional layers to transfer position and channel information respectively. We further propose cross-modal interaction at the attention level to obtain cross-modal attention. Experiments show that our Tmamba achieves promising results in multiple fusion tasks, including infrared-visible image fusion and medical image fusion. Code with checkpoints will be available after the peer-review process. |
| title | Why mamba is effective? Exploit Linear Transformer-Mamba Network for Multi-Modality Image Fusion |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2409.03223 |