MMBind: Unleashing the Potential of Distributed and Heterogeneous Data for Multimodal Learning in IoT
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929742322597888 |
|---|---|
| author | Ouyang, Xiaomin Wu, Jason Kimura, Tomoyoshi Lin, Yihan Verma, Gunjan Abdelzaher, Tarek Srivastava, Mani |
| author_facet | Ouyang, Xiaomin Wu, Jason Kimura, Tomoyoshi Lin, Yihan Verma, Gunjan Abdelzaher, Tarek Srivastava, Mani |
| contents | Multimodal sensing systems are increasingly prevalent in various real-world applications. Most existing multimodal learning approaches heavily rely on training with a large amount of synchronized, complete multimodal data. However, such a setting is impractical in real-world IoT sensing applications where data is typically collected by distributed nodes with heterogeneous data modalities, and is also rarely labeled. In this paper, we propose MMBind, a new data binding approach for multimodal learning on distributed and heterogeneous IoT data. The key idea of MMBind is to construct a pseudo-paired multimodal dataset for model training by binding data from disparate sources and incomplete modalities through a sufficiently descriptive shared modality. We also propose a weighted contrastive learning approach to handle domain shifts among disparate data, coupled with an adaptive multimodal learning architecture capable of training models with heterogeneous modality combinations. Evaluations on ten real-world multimodal datasets highlight that MMBind outperforms state-of-the-art baselines under varying degrees of data incompleteness and domain shift, and holds promise for advancing multimodal foundation model training in IoT applications\footnote (The source code is available via https://github.com/nesl/multimodal-bind). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_12126 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MMBind: Unleashing the Potential of Distributed and Heterogeneous Data for Multimodal Learning in IoT Ouyang, Xiaomin Wu, Jason Kimura, Tomoyoshi Lin, Yihan Verma, Gunjan Abdelzaher, Tarek Srivastava, Mani Machine Learning Multimodal sensing systems are increasingly prevalent in various real-world applications. Most existing multimodal learning approaches heavily rely on training with a large amount of synchronized, complete multimodal data. However, such a setting is impractical in real-world IoT sensing applications where data is typically collected by distributed nodes with heterogeneous data modalities, and is also rarely labeled. In this paper, we propose MMBind, a new data binding approach for multimodal learning on distributed and heterogeneous IoT data. The key idea of MMBind is to construct a pseudo-paired multimodal dataset for model training by binding data from disparate sources and incomplete modalities through a sufficiently descriptive shared modality. We also propose a weighted contrastive learning approach to handle domain shifts among disparate data, coupled with an adaptive multimodal learning architecture capable of training models with heterogeneous modality combinations. Evaluations on ten real-world multimodal datasets highlight that MMBind outperforms state-of-the-art baselines under varying degrees of data incompleteness and domain shift, and holds promise for advancing multimodal foundation model training in IoT applications\footnote (The source code is available via https://github.com/nesl/multimodal-bind). |
| title | MMBind: Unleashing the Potential of Distributed and Heterogeneous Data for Multimodal Learning in IoT |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2411.12126 |