Moving Off-the-Grid: Scene-Grounded Video Representations
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916475799863296 |
|---|---|
| author | van Steenkiste, Sjoerd Zoran, Daniel Yang, Yi Rubanova, Yulia Kabra, Rishabh Doersch, Carl Gokay, Dilara Heyward, Joseph Pot, Etienne Greff, Klaus Hudson, Drew A. Keck, Thomas Albert Carreira, Joao Dosovitskiy, Alexey Sajjadi, Mehdi S. M. Kipf, Thomas |
| author_facet | van Steenkiste, Sjoerd Zoran, Daniel Yang, Yi Rubanova, Yulia Kabra, Rishabh Doersch, Carl Gokay, Dilara Heyward, Joseph Pot, Etienne Greff, Klaus Hudson, Drew A. Keck, Thomas Albert Carreira, Joao Dosovitskiy, Alexey Sajjadi, Mehdi S. M. Kipf, Thomas |
| contents | Current vision models typically maintain a fixed correspondence between their representation structure and image space. Each layer comprises a set of tokens arranged "on-the-grid," which biases patches or tokens to encode information at a specific spatio(-temporal) location. In this work we present Moving Off-the-Grid (MooG), a self-supervised video representation model that offers an alternative approach, allowing tokens to move "off-the-grid" to better enable them to represent scene elements consistently, even as they move across the image plane through time. By using a combination of cross-attention and positional embeddings we disentangle the representation structure and image structure. We find that a simple self-supervised objective--next frame prediction--trained on video data, results in a set of latent tokens which bind to specific scene structures and track them as they move. We demonstrate the usefulness of MooG's learned representation both qualitatively and quantitatively by training readouts on top of the learned representation on a variety of downstream tasks. We show that MooG can provide a strong foundation for different vision tasks when compared to "on-the-grid" baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_05927 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Moving Off-the-Grid: Scene-Grounded Video Representations van Steenkiste, Sjoerd Zoran, Daniel Yang, Yi Rubanova, Yulia Kabra, Rishabh Doersch, Carl Gokay, Dilara Heyward, Joseph Pot, Etienne Greff, Klaus Hudson, Drew A. Keck, Thomas Albert Carreira, Joao Dosovitskiy, Alexey Sajjadi, Mehdi S. M. Kipf, Thomas Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Current vision models typically maintain a fixed correspondence between their representation structure and image space. Each layer comprises a set of tokens arranged "on-the-grid," which biases patches or tokens to encode information at a specific spatio(-temporal) location. In this work we present Moving Off-the-Grid (MooG), a self-supervised video representation model that offers an alternative approach, allowing tokens to move "off-the-grid" to better enable them to represent scene elements consistently, even as they move across the image plane through time. By using a combination of cross-attention and positional embeddings we disentangle the representation structure and image structure. We find that a simple self-supervised objective--next frame prediction--trained on video data, results in a set of latent tokens which bind to specific scene structures and track them as they move. We demonstrate the usefulness of MooG's learned representation both qualitatively and quantitatively by training readouts on top of the learned representation on a variety of downstream tasks. We show that MooG can provide a strong foundation for different vision tasks when compared to "on-the-grid" baselines. |
| title | Moving Off-the-Grid: Scene-Grounded Video Representations |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2411.05927 |