Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.04671 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917661138485248 |
|---|---|
| author | Varshneya, Saurabh Ledent, Antoine Liznerski, Philipp Balinskyy, Andriy Mehta, Purvanshi Mustafa, Waleed Kloft, Marius |
| author_facet | Varshneya, Saurabh Ledent, Antoine Liznerski, Philipp Balinskyy, Andriy Mehta, Purvanshi Mustafa, Waleed Kloft, Marius |
| contents | Conventional machine learning methods are predominantly designed to predict outcomes based on a single data type. However, practical applications may encompass data of diverse types, such as text, images, and audio. We introduce interpretable tensor fusion (InTense), a multimodal learning method for training neural networks to simultaneously learn multimodal data representations and their interpretable fusion. InTense can separately capture both linear combinations and multiplicative interactions of diverse data types, thereby disentangling higher-order interactions from the individual effects of each modality. InTense provides interpretability out of the box by assigning relevance scores to modalities and their associations. The approach is theoretically grounded and yields meaningful relevance scores on multiple synthetic and real-world datasets. Experiments on six real-world datasets show that InTense outperforms existing state-of-the-art multimodal interpretable approaches in terms of accuracy and interpretability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_04671 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Interpretable Tensor Fusion Varshneya, Saurabh Ledent, Antoine Liznerski, Philipp Balinskyy, Andriy Mehta, Purvanshi Mustafa, Waleed Kloft, Marius Machine Learning Conventional machine learning methods are predominantly designed to predict outcomes based on a single data type. However, practical applications may encompass data of diverse types, such as text, images, and audio. We introduce interpretable tensor fusion (InTense), a multimodal learning method for training neural networks to simultaneously learn multimodal data representations and their interpretable fusion. InTense can separately capture both linear combinations and multiplicative interactions of diverse data types, thereby disentangling higher-order interactions from the individual effects of each modality. InTense provides interpretability out of the box by assigning relevance scores to modalities and their associations. The approach is theoretically grounded and yields meaningful relevance scores on multiple synthetic and real-world datasets. Experiments on six real-world datasets show that InTense outperforms existing state-of-the-art multimodal interpretable approaches in terms of accuracy and interpretability. |
| title | Interpretable Tensor Fusion |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2405.04671 |