Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.06735 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929599468797952 |
|---|---|
| author | Kim, Kai Tsai, Howard Sen, Rajat Das, Abhimanyu Zhou, Zihao Tanpure, Abhishek Luo, Mathew Yu, Rose |
| author_facet | Kim, Kai Tsai, Howard Sen, Rajat Das, Abhimanyu Zhou, Zihao Tanpure, Abhishek Luo, Mathew Yu, Rose |
| contents | Current forecasting approaches are largely unimodal and ignore the rich textual data that often accompany the time series due to lack of well-curated multimodal benchmark dataset. In this work, we develop TimeText Corpus (TTC), a carefully curated, time-aligned text and time dataset for multimodal forecasting. Our dataset is composed of sequences of numbers and text aligned to timestamps, and includes data from two different domains: climate science and healthcare. Our data is a significant contribution to the rare selection of available multimodal datasets. We also propose the Hybrid Multi-Modal Forecaster (Hybrid-MMF), a multimodal LLM that jointly forecasts both text and time series data using shared embeddings. However, contrary to our expectations, our Hybrid-MMF model does not outperform existing baselines in our experiments. This negative result highlights the challenges inherent in multimodal forecasting. Our code and data are available at https://github.com/Rose-STL-Lab/Multimodal_ Forecasting. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_06735 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multi-Modal Forecaster: Jointly Predicting Time Series and Textual Data Kim, Kai Tsai, Howard Sen, Rajat Das, Abhimanyu Zhou, Zihao Tanpure, Abhishek Luo, Mathew Yu, Rose Artificial Intelligence Current forecasting approaches are largely unimodal and ignore the rich textual data that often accompany the time series due to lack of well-curated multimodal benchmark dataset. In this work, we develop TimeText Corpus (TTC), a carefully curated, time-aligned text and time dataset for multimodal forecasting. Our dataset is composed of sequences of numbers and text aligned to timestamps, and includes data from two different domains: climate science and healthcare. Our data is a significant contribution to the rare selection of available multimodal datasets. We also propose the Hybrid Multi-Modal Forecaster (Hybrid-MMF), a multimodal LLM that jointly forecasts both text and time series data using shared embeddings. However, contrary to our expectations, our Hybrid-MMF model does not outperform existing baselines in our experiments. This negative result highlights the challenges inherent in multimodal forecasting. Our code and data are available at https://github.com/Rose-STL-Lab/Multimodal_ Forecasting. |
| title | Multi-Modal Forecaster: Jointly Predicting Time Series and Textual Data |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2411.06735 |