Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2412.14294 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914330133397504 |
|---|---|
| author | Pătrăucean, Viorica He, Xu Owen Heyward, Joseph Zhang, Chuhan Sajjadi, Mehdi S. M. Muraru, George-Cristian Zholus, Artem Karami, Mahdi Goroshin, Ross Chen, Yutian Osindero, Simon Carreira, João Pascanu, Razvan |
| author_facet | Pătrăucean, Viorica He, Xu Owen Heyward, Joseph Zhang, Chuhan Sajjadi, Mehdi S. M. Muraru, George-Cristian Zholus, Artem Karami, Mahdi Goroshin, Ross Chen, Yutian Osindero, Simon Carreira, João Pascanu, Razvan |
| contents | We propose a novel block for \emph{causal} video modelling. It relies on a time-space-channel factorisation with dedicated blocks for each dimension: gated linear recurrent units (LRUs) perform information mixing over time, self-attention layers perform mixing over space, and MLPs over channels. The resulting architecture \emph{TRecViT} is causal and shows strong performance on sparse and dense tasks, trained in supervised or self-supervised regimes, being the first causal video model in the state-space models family. Notably, our model outperforms or is on par with the popular (non-causal) ViViT-L model on large scale video datasets (SSv2, Kinetics400), while having $3\times$ less parameters, $12\times$ smaller memory footprint, and $5\times$ lower FLOPs count than the full self-attention ViViT, with an inference throughput of about 300 frames per second, running comfortably in real-time. When compared with causal transformer-based models (TSM, RViT) and other recurrent models like LSTM, TRecViT obtains state-of-the-art results on the challenging SSv2 dataset. Code and checkpoints are available online https://github.com/google-deepmind/trecvit. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_14294 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TRecViT: A Recurrent Video Transformer Pătrăucean, Viorica He, Xu Owen Heyward, Joseph Zhang, Chuhan Sajjadi, Mehdi S. M. Muraru, George-Cristian Zholus, Artem Karami, Mahdi Goroshin, Ross Chen, Yutian Osindero, Simon Carreira, João Pascanu, Razvan Computer Vision and Pattern Recognition Machine Learning We propose a novel block for \emph{causal} video modelling. It relies on a time-space-channel factorisation with dedicated blocks for each dimension: gated linear recurrent units (LRUs) perform information mixing over time, self-attention layers perform mixing over space, and MLPs over channels. The resulting architecture \emph{TRecViT} is causal and shows strong performance on sparse and dense tasks, trained in supervised or self-supervised regimes, being the first causal video model in the state-space models family. Notably, our model outperforms or is on par with the popular (non-causal) ViViT-L model on large scale video datasets (SSv2, Kinetics400), while having $3\times$ less parameters, $12\times$ smaller memory footprint, and $5\times$ lower FLOPs count than the full self-attention ViViT, with an inference throughput of about 300 frames per second, running comfortably in real-time. When compared with causal transformer-based models (TSM, RViT) and other recurrent models like LSTM, TRecViT obtains state-of-the-art results on the challenging SSv2 dataset. Code and checkpoints are available online https://github.com/google-deepmind/trecvit. |
| title | TRecViT: A Recurrent Video Transformer |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2412.14294 |