Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.15475 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917545210019840 |
|---|---|
| author | Xiong, Jiayu Wang, Jing Xue, Jun Wang, Wanlong Kwan, Jianlong Lyu, Xiaosen Jiang, Zhouqiang |
| author_facet | Xiong, Jiayu Wang, Jing Xue, Jun Wang, Wanlong Kwan, Jianlong Lyu, Xiaosen Jiang, Zhouqiang |
| contents | Multimodal learning aims to preserve as much task-related information as possible from different inputs. However, current fusion designs often distort the feedback loop to feature extractors. Aggressively merging modalities entangles their representations, making the feature extractors fragile to incomplete inputs. Meanwhile, attempting to separate features via auxiliary losses frequently introduces optimization conflicts that distract from the primary task. We propose the Self-Consistent Field Autoencoder (SCFAE) to provide a better path for task gradients. Our method follows the self-consistent field principle to balance task learning with feature organization, thereby minimizing mutual information. We use small autoencoders for each modality to keep information intact. The task loss acts as a driving force to select predictive features. The reconstruction loss acts as a constraint to separate these features into independent subspaces. These dual objectives operate through complementary feature subspaces, thereby mitigating optimization interference. We evaluate SCFAE on audio-visual-text, audio-visual, and image-video benchmarks. Results show that SCFAE handles missing data and unequal input sizes more robustly via a simple structure. Gradient analysis confirms that SCFAE avoids conflicts and maintains stable training dynamics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_15475 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multimodal Fusion via Self-Consistent Task-Gradient Fields Xiong, Jiayu Wang, Jing Xue, Jun Wang, Wanlong Kwan, Jianlong Lyu, Xiaosen Jiang, Zhouqiang Computer Vision and Pattern Recognition Multimodal learning aims to preserve as much task-related information as possible from different inputs. However, current fusion designs often distort the feedback loop to feature extractors. Aggressively merging modalities entangles their representations, making the feature extractors fragile to incomplete inputs. Meanwhile, attempting to separate features via auxiliary losses frequently introduces optimization conflicts that distract from the primary task. We propose the Self-Consistent Field Autoencoder (SCFAE) to provide a better path for task gradients. Our method follows the self-consistent field principle to balance task learning with feature organization, thereby minimizing mutual information. We use small autoencoders for each modality to keep information intact. The task loss acts as a driving force to select predictive features. The reconstruction loss acts as a constraint to separate these features into independent subspaces. These dual objectives operate through complementary feature subspaces, thereby mitigating optimization interference. We evaluate SCFAE on audio-visual-text, audio-visual, and image-video benchmarks. Results show that SCFAE handles missing data and unequal input sizes more robustly via a simple structure. Gradient analysis confirms that SCFAE avoids conflicts and maintains stable training dynamics. |
| title | Multimodal Fusion via Self-Consistent Task-Gradient Fields |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.15475 |