Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.01210 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910891614666752 |
|---|---|
| author | Wu, Guanyao Liu, Haoyu Fu, Hongming Peng, Yichuan Liu, Jinyuan Fan, Xin Liu, Risheng |
| author_facet | Wu, Guanyao Liu, Haoyu Fu, Hongming Peng, Yichuan Liu, Jinyuan Fan, Xin Liu, Risheng |
| contents | Multi-modality image fusion, particularly infrared and visible, plays a crucial role in integrating diverse modalities to enhance scene understanding. Although early research prioritized visual quality, preserving fine details and adapting to downstream tasks remains challenging. Recent approaches attempt task-specific design but rarely achieve "The Best of Both Worlds" due to inconsistent optimization goals. To address these issues, we propose a novel method that leverages the semantic knowledge from the Segment Anything Model (SAM) to Grow the quality of fusion results and Enable downstream task adaptability, namely SAGE. Specifically, we design a Semantic Persistent Attention (SPA) Module that efficiently maintains source information via the persistent repository while extracting high-level semantic priors from SAM. More importantly, to eliminate the impractical dependence on SAM during inference, we introduce a bi-level optimization-driven distillation mechanism with triplet losses, which allow the student network to effectively extract knowledge. Extensive experiments show that our method achieves a balance between high-quality visual results and downstream task adaptability while maintaining practical deployment efficiency. The code is available at https://github.com/RollingPlain/SAGE_IVIF. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01210 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyond Wu, Guanyao Liu, Haoyu Fu, Hongming Peng, Yichuan Liu, Jinyuan Fan, Xin Liu, Risheng Computer Vision and Pattern Recognition Multi-modality image fusion, particularly infrared and visible, plays a crucial role in integrating diverse modalities to enhance scene understanding. Although early research prioritized visual quality, preserving fine details and adapting to downstream tasks remains challenging. Recent approaches attempt task-specific design but rarely achieve "The Best of Both Worlds" due to inconsistent optimization goals. To address these issues, we propose a novel method that leverages the semantic knowledge from the Segment Anything Model (SAM) to Grow the quality of fusion results and Enable downstream task adaptability, namely SAGE. Specifically, we design a Semantic Persistent Attention (SPA) Module that efficiently maintains source information via the persistent repository while extracting high-level semantic priors from SAM. More importantly, to eliminate the impractical dependence on SAM during inference, we introduce a bi-level optimization-driven distillation mechanism with triplet losses, which allow the student network to effectively extract knowledge. Extensive experiments show that our method achieves a balance between high-quality visual results and downstream task adaptability while maintaining practical deployment efficiency. The code is available at https://github.com/RollingPlain/SAGE_IVIF. |
| title | Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyond |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.01210 |