Learning Streaming Video Representation via Multitask Training
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913951869042688 |
|---|---|
| author | Yan, Yibin Xu, Jilan Di, Shangzhe Liu, Yikun Shi, Yudi Chen, Qirui Li, Zeqian Huang, Yifei Xie, Weidi |
| author_facet | Yan, Yibin Xu, Jilan Di, Shangzhe Liu, Yikun Shi, Yudi Chen, Qirui Li, Zeqian Huang, Yifei Xie, Weidi |
| contents | Understanding continuous video streams plays a fundamental role in real-time applications including embodied AI and autonomous driving. Unlike offline video understanding, streaming video understanding requires the ability to process video streams frame by frame, preserve historical information, and make low-latency decisions. To address these challenges, our main contributions are three-fold. (i) We develop a novel streaming video backbone, termed as StreamFormer, by incorporating causal temporal attention into a pre-trained vision transformer. This enables efficient streaming video processing while maintaining image representation capability. (ii) To train StreamFormer, we propose to unify diverse spatial-temporal video understanding tasks within a multitask visual-language alignment framework. Hence, StreamFormer learns global semantics, temporal dynamics, and fine-grained spatial relationships simultaneously. (iii) We conduct extensive experiments on online action detection, online video instance segmentation, and video question answering. StreamFormer achieves competitive results while maintaining efficiency, demonstrating its potential for real-time applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_20041 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Learning Streaming Video Representation via Multitask Training Yan, Yibin Xu, Jilan Di, Shangzhe Liu, Yikun Shi, Yudi Chen, Qirui Li, Zeqian Huang, Yifei Xie, Weidi Computer Vision and Pattern Recognition Understanding continuous video streams plays a fundamental role in real-time applications including embodied AI and autonomous driving. Unlike offline video understanding, streaming video understanding requires the ability to process video streams frame by frame, preserve historical information, and make low-latency decisions. To address these challenges, our main contributions are three-fold. (i) We develop a novel streaming video backbone, termed as StreamFormer, by incorporating causal temporal attention into a pre-trained vision transformer. This enables efficient streaming video processing while maintaining image representation capability. (ii) To train StreamFormer, we propose to unify diverse spatial-temporal video understanding tasks within a multitask visual-language alignment framework. Hence, StreamFormer learns global semantics, temporal dynamics, and fine-grained spatial relationships simultaneously. (iii) We conduct extensive experiments on online action detection, online video instance segmentation, and video question answering. StreamFormer achieves competitive results while maintaining efficiency, demonstrating its potential for real-time applications. |
| title | Learning Streaming Video Representation via Multitask Training |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2504.20041 |