InfoMAE: Pair-Efficient Cross-Modal Alignment for Multimodal Time-Series Sensing Signals
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915240641298432 |
|---|---|
| author | Kimura, Tomoyoshi Li, Xinlin Hanna, Osama Chen, Yatong Chen, Yizhuo Kara, Denizhan Wang, Tianshi Li, Jinyang Ouyang, Xiaomin Liu, Shengzhong Srivastava, Mani Diggavi, Suhas Abdelzaher, Tarek |
| author_facet | Kimura, Tomoyoshi Li, Xinlin Hanna, Osama Chen, Yatong Chen, Yizhuo Kara, Denizhan Wang, Tianshi Li, Jinyang Ouyang, Xiaomin Liu, Shengzhong Srivastava, Mani Diggavi, Suhas Abdelzaher, Tarek |
| contents | Standard multimodal self-supervised learning (SSL) algorithms regard cross-modal synchronization as implicit supervisory labels during pretraining, thus posing high requirements on the scale and quality of multimodal samples. These constraints significantly limit the performance of sensing intelligence in IoT applications, as the heterogeneity and the non-interpretability of time-series signals result in abundant unimodal data but scarce high-quality multimodal pairs. This paper proposes InfoMAE, a cross-modal alignment framework that tackles the challenge of multimodal pair efficiency under the SSL setting by facilitating efficient cross-modal alignment of pretrained unimodal representations. InfoMAE achieves \textit{efficient cross-modal alignment} with \textit{limited data pairs} through a novel information theory-inspired formulation that simultaneously addresses distribution-level and instance-level alignment. Extensive experiments on two real-world IoT applications are performed to evaluate InfoMAE's pairing efficiency to bridge pretrained unimodal models into a cohesive joint multimodal model. InfoMAE enhances downstream multimodal tasks by over 60% with significantly improved multimodal pairing efficiency. It also improves unimodal task accuracy by an average of 22%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_09707 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | InfoMAE: Pair-Efficient Cross-Modal Alignment for Multimodal Time-Series Sensing Signals Kimura, Tomoyoshi Li, Xinlin Hanna, Osama Chen, Yatong Chen, Yizhuo Kara, Denizhan Wang, Tianshi Li, Jinyang Ouyang, Xiaomin Liu, Shengzhong Srivastava, Mani Diggavi, Suhas Abdelzaher, Tarek Artificial Intelligence Information Theory Machine Learning Multimedia Standard multimodal self-supervised learning (SSL) algorithms regard cross-modal synchronization as implicit supervisory labels during pretraining, thus posing high requirements on the scale and quality of multimodal samples. These constraints significantly limit the performance of sensing intelligence in IoT applications, as the heterogeneity and the non-interpretability of time-series signals result in abundant unimodal data but scarce high-quality multimodal pairs. This paper proposes InfoMAE, a cross-modal alignment framework that tackles the challenge of multimodal pair efficiency under the SSL setting by facilitating efficient cross-modal alignment of pretrained unimodal representations. InfoMAE achieves \textit{efficient cross-modal alignment} with \textit{limited data pairs} through a novel information theory-inspired formulation that simultaneously addresses distribution-level and instance-level alignment. Extensive experiments on two real-world IoT applications are performed to evaluate InfoMAE's pairing efficiency to bridge pretrained unimodal models into a cohesive joint multimodal model. InfoMAE enhances downstream multimodal tasks by over 60% with significantly improved multimodal pairing efficiency. It also improves unimodal task accuracy by an average of 22%. |
| title | InfoMAE: Pair-Efficient Cross-Modal Alignment for Multimodal Time-Series Sensing Signals |
| topic | Artificial Intelligence Information Theory Machine Learning Multimedia |
| url | https://arxiv.org/abs/2504.09707 |