PastNet: Introducing Physical Inductive Biases for Spatio-temporal Video Prediction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929674404233216 |
|---|---|
| author | Wu, Hao Xu, Fan Chen, Chong Hua, Xian-Sheng Luo, Xiao Wang, Haixin |
| author_facet | Wu, Hao Xu, Fan Chen, Chong Hua, Xian-Sheng Luo, Xiao Wang, Haixin |
| contents | In this paper, we investigate the challenge of spatio-temporal video prediction task, which involves generating future video frames based on historical spatio-temporal observation streams. Existing approaches typically utilize external information such as semantic maps to improve video prediction accuracy, which often neglect the inherent physical knowledge embedded within videos. Worse still, their high computational costs could impede their applications for high-resolution videos. To address these constraints, we introduce a novel framework called \underline{P}hysics-\underline{a}ssisted \underline{S}patio-\underline{t}emporal \underline{Net}work (PastNet) for high-quality video prediction. The core of PastNet lies in incorporating a spectral convolution operator in the Fourier domain, which efficiently introduces inductive biases from the underlying physical laws. Additionally, we employ a memory bank with the estimated intrinsic dimensionality to discretize local features during the processing of complex spatio-temporal signals, thereby reducing computational costs and facilitating efficient high-resolution video prediction. Extensive experiments on various widely-used spatio-temporal video benchmarks demonstrate the effectiveness and efficiency of the proposed PastNet compared with a range of state-of-the-art methods, particularly in high-resolution scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_11421 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | PastNet: Introducing Physical Inductive Biases for Spatio-temporal Video Prediction Wu, Hao Xu, Fan Chen, Chong Hua, Xian-Sheng Luo, Xiao Wang, Haixin Computer Vision and Pattern Recognition Artificial Intelligence 14J60 (Primary) 14F05, 14J26 (Secondary) I.2.6; I.5 In this paper, we investigate the challenge of spatio-temporal video prediction task, which involves generating future video frames based on historical spatio-temporal observation streams. Existing approaches typically utilize external information such as semantic maps to improve video prediction accuracy, which often neglect the inherent physical knowledge embedded within videos. Worse still, their high computational costs could impede their applications for high-resolution videos. To address these constraints, we introduce a novel framework called \underline{P}hysics-\underline{a}ssisted \underline{S}patio-\underline{t}emporal \underline{Net}work (PastNet) for high-quality video prediction. The core of PastNet lies in incorporating a spectral convolution operator in the Fourier domain, which efficiently introduces inductive biases from the underlying physical laws. Additionally, we employ a memory bank with the estimated intrinsic dimensionality to discretize local features during the processing of complex spatio-temporal signals, thereby reducing computational costs and facilitating efficient high-resolution video prediction. Extensive experiments on various widely-used spatio-temporal video benchmarks demonstrate the effectiveness and efficiency of the proposed PastNet compared with a range of state-of-the-art methods, particularly in high-resolution scenarios. |
| title | PastNet: Introducing Physical Inductive Biases for Spatio-temporal Video Prediction |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence 14J60 (Primary) 14F05, 14J26 (Secondary) I.2.6; I.5 |
| url | https://arxiv.org/abs/2305.11421 |