Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2309.03452 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911974893289472 |
|---|---|
| author | Zhao, Zhuokai Palani, Harish Liu, Tianyi Evans, Lena Toner, Ruth |
| author_facet | Zhao, Zhuokai Palani, Harish Liu, Tianyi Evans, Lena Toner, Ruth |
| contents | Multimodal deep learning, especially vision-language models, have gained significant traction in recent years, greatly improving performance on many downstream tasks, including content moderation and violence detection. However, standard multimodal approaches often assume consistent modalities between training and inference, limiting applications in many real-world use cases, as some modalities may not be available during inference. While existing research mitigates this problem through reconstructing the missing modalities, they unavoidably increase unnecessary computational cost, which could be just as critical, especially for large, deployed infrastructures in industry. To this end, we propose a novel guidance network that promotes knowledge sharing during training, taking advantage of the multimodal representations to train better single-modality models to be used for inference. Real-world experiments in violence detection shows that our proposed framework trains single-modality models that significantly outperform traditionally trained counterparts, while avoiding increases in computational cost for inference. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_03452 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Multimodal Guidance Network for Missing-Modality Inference in Content Moderation Zhao, Zhuokai Palani, Harish Liu, Tianyi Evans, Lena Toner, Ruth Computer Vision and Pattern Recognition Machine Learning Multimodal deep learning, especially vision-language models, have gained significant traction in recent years, greatly improving performance on many downstream tasks, including content moderation and violence detection. However, standard multimodal approaches often assume consistent modalities between training and inference, limiting applications in many real-world use cases, as some modalities may not be available during inference. While existing research mitigates this problem through reconstructing the missing modalities, they unavoidably increase unnecessary computational cost, which could be just as critical, especially for large, deployed infrastructures in industry. To this end, we propose a novel guidance network that promotes knowledge sharing during training, taking advantage of the multimodal representations to train better single-modality models to be used for inference. Real-world experiments in violence detection shows that our proposed framework trains single-modality models that significantly outperform traditionally trained counterparts, while avoiding increases in computational cost for inference. |
| title | Multimodal Guidance Network for Missing-Modality Inference in Content Moderation |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2309.03452 |