Rethinking Multimodal Fusion for Time Series: Auxiliary Modalities Need Constrained Fusion
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911538628001792 |
|---|---|
| author | Lee, Seunghan Seo, Jun Lee, Jaehoon Yoo, Sungdong Kim, Minjae Lim, Tae Yoon Kang, Dongwan Choi, Hwanil Lee, SoonYoung Ahn, Wonbin |
| author_facet | Lee, Seunghan Seo, Jun Lee, Jaehoon Yoo, Sungdong Kim, Minjae Lim, Tae Yoon Kang, Dongwan Choi, Hwanil Lee, SoonYoung Ahn, Wonbin |
| contents | Recent advances in multimodal learning have motivated the integration of auxiliary modalities such as text or vision into time series (TS) forecasting. However, most existing methods provide limited gains, often improving performance only in specific datasets or relying on architecture-specific designs that limit generalization. In this paper, we show that multimodal models with naive fusion strategies (e.g., simple addition or concatenation) often underperform unimodal TS models, which we attribute to the uncontrolled integration of auxiliary modalities which may introduce irrelevant information. Motivated by this observation, we explore various constrained fusion methods designed to control such integration and find that they consistently outperform naive fusion methods. Furthermore, we propose Controlled Fusion Adapter (CFA), a simple plug-in method that enables controlled cross-modal interactions without modifying the TS backbone, integrating only relevant textual information aligned with TS dynamics. CFA employs low-rank adapters to filter irrelevant textual information before fusing it into temporal representations. We conduct over 20K experiments across various datasets and TS/text models, demonstrating the effectiveness of the constrained fusion methods including CFA. Code is publicly available at: https://github.com/seunghan96/cfa/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_22372 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Rethinking Multimodal Fusion for Time Series: Auxiliary Modalities Need Constrained Fusion Lee, Seunghan Seo, Jun Lee, Jaehoon Yoo, Sungdong Kim, Minjae Lim, Tae Yoon Kang, Dongwan Choi, Hwanil Lee, SoonYoung Ahn, Wonbin Machine Learning Artificial Intelligence Recent advances in multimodal learning have motivated the integration of auxiliary modalities such as text or vision into time series (TS) forecasting. However, most existing methods provide limited gains, often improving performance only in specific datasets or relying on architecture-specific designs that limit generalization. In this paper, we show that multimodal models with naive fusion strategies (e.g., simple addition or concatenation) often underperform unimodal TS models, which we attribute to the uncontrolled integration of auxiliary modalities which may introduce irrelevant information. Motivated by this observation, we explore various constrained fusion methods designed to control such integration and find that they consistently outperform naive fusion methods. Furthermore, we propose Controlled Fusion Adapter (CFA), a simple plug-in method that enables controlled cross-modal interactions without modifying the TS backbone, integrating only relevant textual information aligned with TS dynamics. CFA employs low-rank adapters to filter irrelevant textual information before fusing it into temporal representations. We conduct over 20K experiments across various datasets and TS/text models, demonstrating the effectiveness of the constrained fusion methods including CFA. Code is publicly available at: https://github.com/seunghan96/cfa/. |
| title | Rethinking Multimodal Fusion for Time Series: Auxiliary Modalities Need Constrained Fusion |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2603.22372 |