Leveraging multimodal explanatory annotations for video interpretation with Modality Specific Dataset
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912329659056128 |
|---|---|
| author | Ancarani, Elisa Tores, Julie Sassatelli, Lucile Sun, Rémy Wu, Hui-Yin Precioso, Frédéric |
| author_facet | Ancarani, Elisa Tores, Julie Sassatelli, Lucile Sun, Rémy Wu, Hui-Yin Precioso, Frédéric |
| contents | We examine the impact of concept-informed supervision on multimodal video interpretation models using MOByGaze, a dataset containing human-annotated explanatory concepts. We introduce Concept Modality Specific Datasets (CMSDs), which consist of data subsets categorized by the modality (visual, textual, or audio) of annotated concepts. Models trained on CMSDs outperform those using traditional legacy training in both early and late fusion approaches. Notably, this approach enables late fusion models to achieve performance close to that of early fusion models. These findings underscore the importance of modality-specific annotations in developing robust, self-explainable video models and contribute to advancing interpretable multimodal learning in complex video analysis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_11232 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Leveraging multimodal explanatory annotations for video interpretation with Modality Specific Dataset Ancarani, Elisa Tores, Julie Sassatelli, Lucile Sun, Rémy Wu, Hui-Yin Precioso, Frédéric Computer Vision and Pattern Recognition Multimedia We examine the impact of concept-informed supervision on multimodal video interpretation models using MOByGaze, a dataset containing human-annotated explanatory concepts. We introduce Concept Modality Specific Datasets (CMSDs), which consist of data subsets categorized by the modality (visual, textual, or audio) of annotated concepts. Models trained on CMSDs outperform those using traditional legacy training in both early and late fusion approaches. Notably, this approach enables late fusion models to achieve performance close to that of early fusion models. These findings underscore the importance of modality-specific annotations in developing robust, self-explainable video models and contribute to advancing interpretable multimodal learning in complex video analysis. |
| title | Leveraging multimodal explanatory annotations for video interpretation with Modality Specific Dataset |
| topic | Computer Vision and Pattern Recognition Multimedia |
| url | https://arxiv.org/abs/2504.11232 |