Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.08781 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916433703731200 |
|---|---|
| author | Guo, Pinxue Zhao, Zixu Gao, Jianxiong Wu, Chongruo He, Tong Zhang, Zheng Xiao, Tianjun Zhang, Wenqiang |
| author_facet | Guo, Pinxue Zhao, Zixu Gao, Jianxiong Wu, Chongruo He, Tong Zhang, Zheng Xiao, Tianjun Zhang, Wenqiang |
| contents | Video segmentation is essential for advancing robotics and autonomous driving, particularly in open-world settings where continuous perception and object association across video frames are critical. While the Segment Anything Model (SAM) has excelled in static image segmentation, extending its capabilities to video segmentation poses significant challenges. We tackle two major hurdles: a) SAM's embedding limitations in associating objects across frames, and b) granularity inconsistencies in object segmentation. To this end, we introduce VideoSAM, an end-to-end framework designed to address these challenges by improving object tracking and segmentation consistency in dynamic environments. VideoSAM integrates an agglomerated backbone, RADIO, enabling object association through similarity metrics and introduces Cycle-ack-Pairs Propagation with a memory mechanism for stable object tracking. Additionally, we incorporate an autoregressive object-token mechanism within the SAM decoder to maintain consistent granularity across frames. Our method is extensively evaluated on the UVO and BURST benchmarks, and robotic videos from RoboTAP, demonstrating its effectiveness and robustness in real-world scenarios. All codes will be available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_08781 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | VideoSAM: Open-World Video Segmentation Guo, Pinxue Zhao, Zixu Gao, Jianxiong Wu, Chongruo He, Tong Zhang, Zheng Xiao, Tianjun Zhang, Wenqiang Computer Vision and Pattern Recognition Video segmentation is essential for advancing robotics and autonomous driving, particularly in open-world settings where continuous perception and object association across video frames are critical. While the Segment Anything Model (SAM) has excelled in static image segmentation, extending its capabilities to video segmentation poses significant challenges. We tackle two major hurdles: a) SAM's embedding limitations in associating objects across frames, and b) granularity inconsistencies in object segmentation. To this end, we introduce VideoSAM, an end-to-end framework designed to address these challenges by improving object tracking and segmentation consistency in dynamic environments. VideoSAM integrates an agglomerated backbone, RADIO, enabling object association through similarity metrics and introduces Cycle-ack-Pairs Propagation with a memory mechanism for stable object tracking. Additionally, we incorporate an autoregressive object-token mechanism within the SAM decoder to maintain consistent granularity across frames. Our method is extensively evaluated on the UVO and BURST benchmarks, and robotic videos from RoboTAP, demonstrating its effectiveness and robustness in real-world scenarios. All codes will be available. |
| title | VideoSAM: Open-World Video Segmentation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.08781 |