Rethinking Early-Fusion Strategies for Improved Multispectral Object Detection
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916400644227072 |
|---|---|
| author | Zhang, Xue Cao, Si-Yuan Wang, Fang Zhang, Runmin Wu, Zhe Zhang, Xiaohan Bai, Xiaokai Shen, Hui-Liang |
| author_facet | Zhang, Xue Cao, Si-Yuan Wang, Fang Zhang, Runmin Wu, Zhe Zhang, Xiaohan Bai, Xiaokai Shen, Hui-Liang |
| contents | Most recent multispectral object detectors employ a two-branch structure to extract features from RGB and thermal images. While the two-branch structure achieves better performance than a single-branch structure, it overlooks inference efficiency. This conflict is increasingly aggressive, as recent works solely pursue higher performance rather than both performance and efficiency. In this paper, we address this issue by improving the performance of efficient single-branch structures. We revisit the reasons causing the performance gap between these structures. For the first time, we reveal the information interference problem in the naive early-fusion strategy adopted by previous single-branch structures. Besides, we find that the domain gap between multispectral images, and weak feature representation of the single-branch structure are also key obstacles for performance. Focusing on these three problems, we propose corresponding solutions, including a novel shape-priority early-fusion strategy, a weakly supervised learning method, and a core knowledge distillation technique. Experiments demonstrate that single-branch networks equipped with these three contributions achieve significant performance enhancements while retaining high efficiency. Our code will be available at \url{https://github.com/XueZ-phd/Efficient-RGB-T-Early-Fusion-Detection}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_16038 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Rethinking Early-Fusion Strategies for Improved Multispectral Object Detection Zhang, Xue Cao, Si-Yuan Wang, Fang Zhang, Runmin Wu, Zhe Zhang, Xiaohan Bai, Xiaokai Shen, Hui-Liang Computer Vision and Pattern Recognition Most recent multispectral object detectors employ a two-branch structure to extract features from RGB and thermal images. While the two-branch structure achieves better performance than a single-branch structure, it overlooks inference efficiency. This conflict is increasingly aggressive, as recent works solely pursue higher performance rather than both performance and efficiency. In this paper, we address this issue by improving the performance of efficient single-branch structures. We revisit the reasons causing the performance gap between these structures. For the first time, we reveal the information interference problem in the naive early-fusion strategy adopted by previous single-branch structures. Besides, we find that the domain gap between multispectral images, and weak feature representation of the single-branch structure are also key obstacles for performance. Focusing on these three problems, we propose corresponding solutions, including a novel shape-priority early-fusion strategy, a weakly supervised learning method, and a core knowledge distillation technique. Experiments demonstrate that single-branch networks equipped with these three contributions achieve significant performance enhancements while retaining high efficiency. Our code will be available at \url{https://github.com/XueZ-phd/Efficient-RGB-T-Early-Fusion-Detection}. |
| title | Rethinking Early-Fusion Strategies for Improved Multispectral Object Detection |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.16038 |