Learning Robust Anymodal Segmentor with Unimodal and Cross-modal Distillation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909611580194816 |
|---|---|
| author | Zheng, Xu Xue, Haiwei Chen, Jialei Yan, Yibo Jiang, Lutao Lyu, Yuanhuiyi Yang, Kailun Zhang, Linfeng Hu, Xuming |
| author_facet | Zheng, Xu Xue, Haiwei Chen, Jialei Yan, Yibo Jiang, Lutao Lyu, Yuanhuiyi Yang, Kailun Zhang, Linfeng Hu, Xuming |
| contents | Simultaneously using multimodal inputs from multiple sensors to train segmentors is intuitively advantageous but practically challenging. A key challenge is unimodal bias, where multimodal segmentors over rely on certain modalities, causing performance drops when others are missing, common in real world applications. To this end, we develop the first framework for learning robust segmentor that can handle any combinations of visual modalities. Specifically, we first introduce a parallel multimodal learning strategy for learning a strong teacher. The cross-modal and unimodal distillation is then achieved in the multi scale representation space by transferring the feature level knowledge from multimodal to anymodal segmentors, aiming at addressing the unimodal bias and avoiding over-reliance on specific modalities. Moreover, a prediction level modality agnostic semantic distillation is proposed to achieve semantic knowledge transferring for segmentation. Extensive experiments on both synthetic and real-world multi-sensor benchmarks demonstrate that our method achieves superior performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_17141 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Learning Robust Anymodal Segmentor with Unimodal and Cross-modal Distillation Zheng, Xu Xue, Haiwei Chen, Jialei Yan, Yibo Jiang, Lutao Lyu, Yuanhuiyi Yang, Kailun Zhang, Linfeng Hu, Xuming Computer Vision and Pattern Recognition Simultaneously using multimodal inputs from multiple sensors to train segmentors is intuitively advantageous but practically challenging. A key challenge is unimodal bias, where multimodal segmentors over rely on certain modalities, causing performance drops when others are missing, common in real world applications. To this end, we develop the first framework for learning robust segmentor that can handle any combinations of visual modalities. Specifically, we first introduce a parallel multimodal learning strategy for learning a strong teacher. The cross-modal and unimodal distillation is then achieved in the multi scale representation space by transferring the feature level knowledge from multimodal to anymodal segmentors, aiming at addressing the unimodal bias and avoiding over-reliance on specific modalities. Moreover, a prediction level modality agnostic semantic distillation is proposed to achieve semantic knowledge transferring for segmentation. Extensive experiments on both synthetic and real-world multi-sensor benchmarks demonstrate that our method achieves superior performance. |
| title | Learning Robust Anymodal Segmentor with Unimodal and Cross-modal Distillation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2411.17141 |