Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908677938610176 |
|---|---|
| author | Karypidis, Efstathios Kakogeorgiou, Ioannis Gidaris, Spyros Komodakis, Nikos |
| author_facet | Karypidis, Efstathios Kakogeorgiou, Ioannis Gidaris, Spyros Komodakis, Nikos |
| contents | Semantic future prediction is important for autonomous systems navigating dynamic environments. This paper introduces FUTURIST, a method for multimodal future semantic prediction that uses a unified and efficient visual sequence transformer architecture. Our approach incorporates a multimodal masked visual modeling objective and a novel masking mechanism designed for multimodal training. This allows the model to effectively integrate visible information from various modalities, improving prediction accuracy. Additionally, we propose a VAE-free hierarchical tokenization process, which reduces computational complexity, streamlines the training pipeline, and enables end-to-end training with high-resolution, multimodal inputs. We validate FUTURIST on the Cityscapes dataset, demonstrating state-of-the-art performance in future semantic segmentation for both short- and mid-term forecasting. Project page and code at https://futurist-cvpr2025.github.io/ . |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_08303 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers Karypidis, Efstathios Kakogeorgiou, Ioannis Gidaris, Spyros Komodakis, Nikos Computer Vision and Pattern Recognition Semantic future prediction is important for autonomous systems navigating dynamic environments. This paper introduces FUTURIST, a method for multimodal future semantic prediction that uses a unified and efficient visual sequence transformer architecture. Our approach incorporates a multimodal masked visual modeling objective and a novel masking mechanism designed for multimodal training. This allows the model to effectively integrate visible information from various modalities, improving prediction accuracy. Additionally, we propose a VAE-free hierarchical tokenization process, which reduces computational complexity, streamlines the training pipeline, and enables end-to-end training with high-resolution, multimodal inputs. We validate FUTURIST on the Cityscapes dataset, demonstrating state-of-the-art performance in future semantic segmentation for both short- and mid-term forecasting. Project page and code at https://futurist-cvpr2025.github.io/ . |
| title | Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2501.08303 |