UnSAMFlow: Unsupervised Optical Flow Guided by Segment Anything Model
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929335491887104 |
|---|---|
| author | Yuan, Shuai Luo, Lei Hui, Zhuo Pu, Can Xiang, Xiaoyu Ranjan, Rakesh Demandolx, Denis |
| author_facet | Yuan, Shuai Luo, Lei Hui, Zhuo Pu, Can Xiang, Xiaoyu Ranjan, Rakesh Demandolx, Denis |
| contents | Traditional unsupervised optical flow methods are vulnerable to occlusions and motion boundaries due to lack of object-level information. Therefore, we propose UnSAMFlow, an unsupervised flow network that also leverages object information from the latest foundation model Segment Anything Model (SAM). We first include a self-supervised semantic augmentation module tailored to SAM masks. We also analyze the poor gradient landscapes of traditional smoothness losses and propose a new smoothness definition based on homography instead. A simple yet effective mask feature module has also been added to further aggregate features on the object level. With all these adaptations, our method produces clear optical flow estimation with sharp boundaries around objects, which outperforms state-of-the-art methods on both KITTI and Sintel datasets. Our method also generalizes well across domains and runs very efficiently. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_02608 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | UnSAMFlow: Unsupervised Optical Flow Guided by Segment Anything Model Yuan, Shuai Luo, Lei Hui, Zhuo Pu, Can Xiang, Xiaoyu Ranjan, Rakesh Demandolx, Denis Computer Vision and Pattern Recognition Artificial Intelligence Robotics Traditional unsupervised optical flow methods are vulnerable to occlusions and motion boundaries due to lack of object-level information. Therefore, we propose UnSAMFlow, an unsupervised flow network that also leverages object information from the latest foundation model Segment Anything Model (SAM). We first include a self-supervised semantic augmentation module tailored to SAM masks. We also analyze the poor gradient landscapes of traditional smoothness losses and propose a new smoothness definition based on homography instead. A simple yet effective mask feature module has also been added to further aggregate features on the object level. With all these adaptations, our method produces clear optical flow estimation with sharp boundaries around objects, which outperforms state-of-the-art methods on both KITTI and Sintel datasets. Our method also generalizes well across domains and runs very efficiently. |
| title | UnSAMFlow: Unsupervised Optical Flow Guided by Segment Anything Model |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Robotics |
| url | https://arxiv.org/abs/2405.02608 |