Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.23287 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912524029394944 |
|---|---|
| author | Bagchi, Anurag Bao, Zhipeng Wang, Yu-Xiong Tokmakov, Pavel Hebert, Martial |
| author_facet | Bagchi, Anurag Bao, Zhipeng Wang, Yu-Xiong Tokmakov, Pavel Hebert, Martial |
| contents | We present REM, a framework for segmenting a wide range of concepts in video that can be described through natural language. Our method leverages the universal visual-language mapping learned by video diffusion models on Internet-scale data by fine-tuning them on small-scale Referring Object Segmentation datasets. Our key insight is to preserve the entirety of the generative model's architecture by shifting its objective from predicting noise to predicting mask latents. The resulting model can accurately segment rare and unseen objects, despite only being trained on a limited set of categories. Additionally, it can effortlessly generalize to non-object dynamic concepts, such as smoke or raindrops, as demonstrated in our new benchmark for Referring Video Process Segmentation (Ref-VPS). REM performs on par with the state-of-the-art on in-domain datasets, like Ref-DAVIS, while outperforming them by up to 12 IoU points out-of-domain, leveraging the power of generative pre-training. We also show that advancements in video generation directly improve segmentation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_23287 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ReferEverything: Towards Segmenting Everything We Can Speak of in Videos Bagchi, Anurag Bao, Zhipeng Wang, Yu-Xiong Tokmakov, Pavel Hebert, Martial Computer Vision and Pattern Recognition We present REM, a framework for segmenting a wide range of concepts in video that can be described through natural language. Our method leverages the universal visual-language mapping learned by video diffusion models on Internet-scale data by fine-tuning them on small-scale Referring Object Segmentation datasets. Our key insight is to preserve the entirety of the generative model's architecture by shifting its objective from predicting noise to predicting mask latents. The resulting model can accurately segment rare and unseen objects, despite only being trained on a limited set of categories. Additionally, it can effortlessly generalize to non-object dynamic concepts, such as smoke or raindrops, as demonstrated in our new benchmark for Referring Video Process Segmentation (Ref-VPS). REM performs on par with the state-of-the-art on in-domain datasets, like Ref-DAVIS, while outperforming them by up to 12 IoU points out-of-domain, leveraging the power of generative pre-training. We also show that advancements in video generation directly improve segmentation. |
| title | ReferEverything: Towards Segmenting Everything We Can Speak of in Videos |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.23287 |