Learning from One Continuous Video Stream
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914732723666944 |
|---|---|
| author | Carreira, João King, Michael Pătrăucean, Viorica Gokay, Dilara Ionescu, Cătălin Yang, Yi Zoran, Daniel Heyward, Joseph Doersch, Carl Aytar, Yusuf Damen, Dima Zisserman, Andrew |
| author_facet | Carreira, João King, Michael Pătrăucean, Viorica Gokay, Dilara Ionescu, Cătălin Yang, Yi Zoran, Daniel Heyward, Joseph Doersch, Carl Aytar, Yusuf Damen, Dima Zisserman, Andrew |
| contents | We introduce a framework for online learning from a single continuous video stream -- the way people and animals learn, without mini-batches, data augmentation or shuffling. This poses great challenges given the high correlation between consecutive video frames and there is very little prior work on it. Our framework allows us to do a first deep dive into the topic and includes a collection of streams and tasks composed from two existing video datasets, plus methodology for performance evaluation that considers both adaptation and generalization. We employ pixel-to-pixel modelling as a practical and flexible way to switch between pre-training and single-stream evaluation as well as between arbitrary tasks, without ever requiring changes to models and always using the same pixel loss. Equipped with this framework we obtained large single-stream learning gains from pre-training with a novel family of future prediction tasks, found that momentum hurts, and that the pace of weight updates matters. The combination of these insights leads to matching the performance of IID learning with batch size 1, when using the same architecture and without costly replay buffers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_00598 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Learning from One Continuous Video Stream Carreira, João King, Michael Pătrăucean, Viorica Gokay, Dilara Ionescu, Cătălin Yang, Yi Zoran, Daniel Heyward, Joseph Doersch, Carl Aytar, Yusuf Damen, Dima Zisserman, Andrew Computer Vision and Pattern Recognition Artificial Intelligence We introduce a framework for online learning from a single continuous video stream -- the way people and animals learn, without mini-batches, data augmentation or shuffling. This poses great challenges given the high correlation between consecutive video frames and there is very little prior work on it. Our framework allows us to do a first deep dive into the topic and includes a collection of streams and tasks composed from two existing video datasets, plus methodology for performance evaluation that considers both adaptation and generalization. We employ pixel-to-pixel modelling as a practical and flexible way to switch between pre-training and single-stream evaluation as well as between arbitrary tasks, without ever requiring changes to models and always using the same pixel loss. Equipped with this framework we obtained large single-stream learning gains from pre-training with a novel family of future prediction tasks, found that momentum hurts, and that the pace of weight updates matters. The combination of these insights leads to matching the performance of IID learning with batch size 1, when using the same architecture and without costly replay buffers. |
| title | Learning from One Continuous Video Stream |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2312.00598 |