TNF: Tri-branch Neural Fusion for Multimodal Medical Data Classification
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909133352992768 |
|---|---|
| author | Zheng, Tong Sone, Shusaku Ushiku, Yoshitaka Oba, Yuki Ma, Jiaxin |
| author_facet | Zheng, Tong Sone, Shusaku Ushiku, Yoshitaka Oba, Yuki Ma, Jiaxin |
| contents | This paper presents a Tri-branch Neural Fusion (TNF) approach designed for classifying multimodal medical images and tabular data. It also introduces two solutions to address the challenge of label inconsistency in multimodal classification. Traditional methods in multi-modality medical data classification often rely on single-label approaches, typically merging features from two distinct input modalities. This becomes problematic when features are mutually exclusive or labels differ across modalities, leading to reduced accuracy. To overcome this, our TNF approach implements a tri-branch framework that manages three separate outputs: one for image modality, another for tabular modality, and a third hybrid output that fuses both image and tabular data. The final decision is made through an ensemble method that integrates likelihoods from all three branches. We validate the effectiveness of TNF through extensive experiments, which illustrate its superiority over traditional fusion and ensemble methods in various convolutional neural networks and transformer-based architectures across multiple datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_01802 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TNF: Tri-branch Neural Fusion for Multimodal Medical Data Classification Zheng, Tong Sone, Shusaku Ushiku, Yoshitaka Oba, Yuki Ma, Jiaxin Computer Vision and Pattern Recognition This paper presents a Tri-branch Neural Fusion (TNF) approach designed for classifying multimodal medical images and tabular data. It also introduces two solutions to address the challenge of label inconsistency in multimodal classification. Traditional methods in multi-modality medical data classification often rely on single-label approaches, typically merging features from two distinct input modalities. This becomes problematic when features are mutually exclusive or labels differ across modalities, leading to reduced accuracy. To overcome this, our TNF approach implements a tri-branch framework that manages three separate outputs: one for image modality, another for tabular modality, and a third hybrid output that fuses both image and tabular data. The final decision is made through an ensemble method that integrates likelihoods from all three branches. We validate the effectiveness of TNF through extensive experiments, which illustrate its superiority over traditional fusion and ensemble methods in various convolutional neural networks and transformer-based architectures across multiple datasets. |
| title | TNF: Tri-branch Neural Fusion for Multimodal Medical Data Classification |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.01802 |