MambaTrans: Multimodal Fusion Image Translation via Large Language Model Priors for Downstream Visual Tasks
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909732116103168 |
|---|---|
| author | Xu, Yushen Li, Xiaosong Kuang, Zhenyu Cheng, Xiaoqi Tan, Haishu Li, Huafeng |
| author_facet | Xu, Yushen Li, Xiaosong Kuang, Zhenyu Cheng, Xiaoqi Tan, Haishu Li, Huafeng |
| contents | The goal of multimodal image fusion is to integrate complementary information from infrared and visible images, generating multimodal fused images for downstream tasks. Existing downstream pre-training models are typically trained on visible images. However, the significant pixel distribution differences between visible and multimodal fusion images can degrade downstream task performance, sometimes even below that of using only visible images. This paper explores adapting multimodal fused images with significant modality differences to object detection and semantic segmentation models trained on visible images. To address this, we propose MambaTrans, a novel multimodal fusion image modality translator. MambaTrans uses descriptions from a multimodal large language model and masks from semantic segmentation models as input. Its core component, the Multi-Model State Space Block, combines mask-image-text cross-attention and a 3D-Selective Scan Module, enhancing pure visual capabilities. By leveraging object detection prior knowledge, MambaTrans minimizes detection loss during training and captures long-term dependencies among text, masks, and images. This enables favorable results in pre-trained models without adjusting their parameters. Experiments on public datasets show that MambaTrans effectively improves multimodal image performance in downstream tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_07803 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MambaTrans: Multimodal Fusion Image Translation via Large Language Model Priors for Downstream Visual Tasks Xu, Yushen Li, Xiaosong Kuang, Zhenyu Cheng, Xiaoqi Tan, Haishu Li, Huafeng Computer Vision and Pattern Recognition The goal of multimodal image fusion is to integrate complementary information from infrared and visible images, generating multimodal fused images for downstream tasks. Existing downstream pre-training models are typically trained on visible images. However, the significant pixel distribution differences between visible and multimodal fusion images can degrade downstream task performance, sometimes even below that of using only visible images. This paper explores adapting multimodal fused images with significant modality differences to object detection and semantic segmentation models trained on visible images. To address this, we propose MambaTrans, a novel multimodal fusion image modality translator. MambaTrans uses descriptions from a multimodal large language model and masks from semantic segmentation models as input. Its core component, the Multi-Model State Space Block, combines mask-image-text cross-attention and a 3D-Selective Scan Module, enhancing pure visual capabilities. By leveraging object detection prior knowledge, MambaTrans minimizes detection loss during training and captures long-term dependencies among text, masks, and images. This enables favorable results in pre-trained models without adjusting their parameters. Experiments on public datasets show that MambaTrans effectively improves multimodal image performance in downstream tasks. |
| title | MambaTrans: Multimodal Fusion Image Translation via Large Language Model Priors for Downstream Visual Tasks |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2508.07803 |