Balancing Multimodal Training Through Game-Theoretic Regularization
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916980851736576 |
|---|---|
| author | Kontras, Konstantinos Strypsteen, Thomas Chatzichristos, Christos Liang, Paul Pu Blaschko, Matthew De Vos, Maarten |
| author_facet | Kontras, Konstantinos Strypsteen, Thomas Chatzichristos, Christos Liang, Paul Pu Blaschko, Matthew De Vos, Maarten |
| contents | Multimodal learning holds promise for richer information extraction by capturing dependencies across data sources. Yet, current training methods often underperform due to modality competition, a phenomenon where modalities contend for training resources leaving some underoptimized. This raises a pivotal question: how can we address training imbalances, ensure adequate optimization across all modalities, and achieve consistent performance improvements as we transition from unimodal to multimodal data? This paper proposes the Multimodal Competition Regularizer (MCR), inspired by a mutual information (MI) decomposition designed to prevent the adverse effects of competition in multimodal training. Our key contributions are: 1) A game-theoretic framework that adaptively balances modality contributions by encouraging each to maximize its informative role in the final prediction 2) Refining lower and upper bounds for each MI term to enhance the extraction of both task-relevant unique and shared information across modalities. 3) Proposing latent space permutations for conditional MI estimation, significantly improving computational efficiency. MCR outperforms all previously suggested training strategies and simple baseline, clearly demonstrating that training modalities jointly leads to important performance gains on both synthetic and large real-world datasets. We release our code and models at https://github.com/kkontras/MCR. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_07335 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Balancing Multimodal Training Through Game-Theoretic Regularization Kontras, Konstantinos Strypsteen, Thomas Chatzichristos, Christos Liang, Paul Pu Blaschko, Matthew De Vos, Maarten Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Computer Science and Game Theory Multimedia Multimodal learning holds promise for richer information extraction by capturing dependencies across data sources. Yet, current training methods often underperform due to modality competition, a phenomenon where modalities contend for training resources leaving some underoptimized. This raises a pivotal question: how can we address training imbalances, ensure adequate optimization across all modalities, and achieve consistent performance improvements as we transition from unimodal to multimodal data? This paper proposes the Multimodal Competition Regularizer (MCR), inspired by a mutual information (MI) decomposition designed to prevent the adverse effects of competition in multimodal training. Our key contributions are: 1) A game-theoretic framework that adaptively balances modality contributions by encouraging each to maximize its informative role in the final prediction 2) Refining lower and upper bounds for each MI term to enhance the extraction of both task-relevant unique and shared information across modalities. 3) Proposing latent space permutations for conditional MI estimation, significantly improving computational efficiency. MCR outperforms all previously suggested training strategies and simple baseline, clearly demonstrating that training modalities jointly leads to important performance gains on both synthetic and large real-world datasets. We release our code and models at https://github.com/kkontras/MCR. |
| title | Balancing Multimodal Training Through Game-Theoretic Regularization |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Computer Science and Game Theory Multimedia |
| url | https://arxiv.org/abs/2411.07335 |