E2E-MFD: Towards End-to-End Synchronous Multimodal Fusion Detection
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916582622494720 |
|---|---|
| author | Zhang, Jiaqing Cao, Mingxiang Xie, Weiying Lei, Jie Li, Daixun Huang, Wenbo Li, Yunsong Yang, Xue |
| author_facet | Zhang, Jiaqing Cao, Mingxiang Xie, Weiying Lei, Jie Li, Daixun Huang, Wenbo Li, Yunsong Yang, Xue |
| contents | Multimodal image fusion and object detection are crucial for autonomous driving. While current methods have advanced the fusion of texture details and semantic information, their complex training processes hinder broader applications. Addressing this challenge, we introduce E2E-MFD, a novel end-to-end algorithm for multimodal fusion detection. E2E-MFD streamlines the process, achieving high performance with a single training phase. It employs synchronous joint optimization across components to avoid suboptimal solutions tied to individual tasks. Furthermore, it implements a comprehensive optimization strategy in the gradient matrix for shared parameters, ensuring convergence to an optimal fusion detection configuration. Our extensive testing on multiple public datasets reveals E2E-MFD's superior capabilities, showcasing not only visually appealing image fusion but also impressive detection outcomes, such as a 3.9% and 2.0% mAP50 increase on horizontal object detection dataset M3FD and oriented object detection dataset DroneVehicle, respectively, compared to state-of-the-art approaches. The code is released at https://github.com/icey-zhang/E2E-MFD. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_09323 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | E2E-MFD: Towards End-to-End Synchronous Multimodal Fusion Detection Zhang, Jiaqing Cao, Mingxiang Xie, Weiying Lei, Jie Li, Daixun Huang, Wenbo Li, Yunsong Yang, Xue Computer Vision and Pattern Recognition Multimodal image fusion and object detection are crucial for autonomous driving. While current methods have advanced the fusion of texture details and semantic information, their complex training processes hinder broader applications. Addressing this challenge, we introduce E2E-MFD, a novel end-to-end algorithm for multimodal fusion detection. E2E-MFD streamlines the process, achieving high performance with a single training phase. It employs synchronous joint optimization across components to avoid suboptimal solutions tied to individual tasks. Furthermore, it implements a comprehensive optimization strategy in the gradient matrix for shared parameters, ensuring convergence to an optimal fusion detection configuration. Our extensive testing on multiple public datasets reveals E2E-MFD's superior capabilities, showcasing not only visually appealing image fusion but also impressive detection outcomes, such as a 3.9% and 2.0% mAP50 increase on horizontal object detection dataset M3FD and oriented object detection dataset DroneVehicle, respectively, compared to state-of-the-art approaches. The code is released at https://github.com/icey-zhang/E2E-MFD. |
| title | E2E-MFD: Towards End-to-End Synchronous Multimodal Fusion Detection |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.09323 |