Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.15209 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929664283377664 |
|---|---|
| author | Kim, Taeheon Chung, Sangyun Yeom, Damin Yu, Youngjoon Kim, Hak Gu Ro, Yong Man |
| author_facet | Kim, Taeheon Chung, Sangyun Yeom, Damin Yu, Youngjoon Kim, Hak Gu Ro, Yong Man |
| contents | Multispectral pedestrian detection is attractive for around-the-clock applications due to the complementary information between RGB and thermal modalities. However, current models often fail to detect pedestrians in certain cases (e.g., thermal-obscured pedestrians), particularly due to the modality bias learned from statistically biased datasets. In this paper, we investigate how to mitigate modality bias in multispectral pedestrian detection using Large Language Models (LLMs). Accordingly, we design a Multispectral Chain-of-Thought (MSCoT) prompting strategy, which prompts the LLM to perform multispectral pedestrian detection. Moreover, we propose a novel Multispectral Chain-of-Thought Detection (MSCoTDet) framework that integrates MSCoT prompting into multispectral pedestrian detection. To this end, we design a Language-driven Multi-modal Fusion (LMF) strategy that enables fusing the outputs of MSCoT prompting with the detection results of vision-based multispectral pedestrian detection models. Extensive experiments validate that MSCoTDet effectively mitigates modality biases and improves multispectral pedestrian detection. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_15209 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MSCoTDet: Language-driven Multi-modal Fusion for Improved Multispectral Pedestrian Detection Kim, Taeheon Chung, Sangyun Yeom, Damin Yu, Youngjoon Kim, Hak Gu Ro, Yong Man Computer Vision and Pattern Recognition Multispectral pedestrian detection is attractive for around-the-clock applications due to the complementary information between RGB and thermal modalities. However, current models often fail to detect pedestrians in certain cases (e.g., thermal-obscured pedestrians), particularly due to the modality bias learned from statistically biased datasets. In this paper, we investigate how to mitigate modality bias in multispectral pedestrian detection using Large Language Models (LLMs). Accordingly, we design a Multispectral Chain-of-Thought (MSCoT) prompting strategy, which prompts the LLM to perform multispectral pedestrian detection. Moreover, we propose a novel Multispectral Chain-of-Thought Detection (MSCoTDet) framework that integrates MSCoT prompting into multispectral pedestrian detection. To this end, we design a Language-driven Multi-modal Fusion (LMF) strategy that enables fusing the outputs of MSCoT prompting with the detection results of vision-based multispectral pedestrian detection models. Extensive experiments validate that MSCoTDet effectively mitigates modality biases and improves multispectral pedestrian detection. |
| title | MSCoTDet: Language-driven Multi-modal Fusion for Improved Multispectral Pedestrian Detection |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.15209 |