UniSTD: Towards Unified Spatio-Temporal Learning across Diverse Disciplines
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912295503790080 |
|---|---|
| author | Tang, Chen Ma, Xinzhu Su, Encheng Song, Xiufeng Liu, Xiaohong Li, Wei-Hong Bai, Lei Ouyang, Wanli Yue, Xiangyu |
| author_facet | Tang, Chen Ma, Xinzhu Su, Encheng Song, Xiufeng Liu, Xiaohong Li, Wei-Hong Bai, Lei Ouyang, Wanli Yue, Xiangyu |
| contents | Traditional spatiotemporal models generally rely on task-specific architectures, which limit their generalizability and scalability across diverse tasks due to domain-specific design requirements. In this paper, we introduce \textbf{UniSTD}, a unified Transformer-based framework for spatiotemporal modeling, which is inspired by advances in recent foundation models with the two-stage pretraining-then-adaption paradigm. Specifically, our work demonstrates that task-agnostic pretraining on 2D vision and vision-text datasets can build a generalizable model foundation for spatiotemporal learning, followed by specialized joint training on spatiotemporal datasets to enhance task-specific adaptability. To improve the learning capabilities across domains, our framework employs a rank-adaptive mixture-of-expert adaptation by using fractional interpolation to relax the discrete variables so that can be optimized in the continuous space. Additionally, we introduce a temporal module to incorporate temporal dynamics explicitly. We evaluate our approach on a large-scale dataset covering 10 tasks across 4 disciplines, demonstrating that a unified spatiotemporal model can achieve scalable, cross-task learning and support up to 10 tasks simultaneously within one model while reducing training costs in multi-domain applications. Code will be available at https://github.com/1hunters/UniSTD. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_20748 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | UniSTD: Towards Unified Spatio-Temporal Learning across Diverse Disciplines Tang, Chen Ma, Xinzhu Su, Encheng Song, Xiufeng Liu, Xiaohong Li, Wei-Hong Bai, Lei Ouyang, Wanli Yue, Xiangyu Computer Vision and Pattern Recognition Traditional spatiotemporal models generally rely on task-specific architectures, which limit their generalizability and scalability across diverse tasks due to domain-specific design requirements. In this paper, we introduce \textbf{UniSTD}, a unified Transformer-based framework for spatiotemporal modeling, which is inspired by advances in recent foundation models with the two-stage pretraining-then-adaption paradigm. Specifically, our work demonstrates that task-agnostic pretraining on 2D vision and vision-text datasets can build a generalizable model foundation for spatiotemporal learning, followed by specialized joint training on spatiotemporal datasets to enhance task-specific adaptability. To improve the learning capabilities across domains, our framework employs a rank-adaptive mixture-of-expert adaptation by using fractional interpolation to relax the discrete variables so that can be optimized in the continuous space. Additionally, we introduce a temporal module to incorporate temporal dynamics explicitly. We evaluate our approach on a large-scale dataset covering 10 tasks across 4 disciplines, demonstrating that a unified spatiotemporal model can achieve scalable, cross-task learning and support up to 10 tasks simultaneously within one model while reducing training costs in multi-domain applications. Code will be available at https://github.com/1hunters/UniSTD. |
| title | UniSTD: Towards Unified Spatio-Temporal Learning across Diverse Disciplines |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.20748 |