Semantic Segmentation on VSPW Dataset through Masked Video Consistency
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917687624466432 |
|---|---|
| author | Liang, Chen Guo, Qiang Yu, Chongkai Wu, Chengjing Liu, Ting Liu, Luoqi |
| author_facet | Liang, Chen Guo, Qiang Yu, Chongkai Wu, Chengjing Liu, Ting Liu, Luoqi |
| contents | Pixel-level Video Understanding requires effectively integrating three-dimensional data in both spatial and temporal dimensions to learn accurate and stable semantic information from continuous frames. However, existing advanced models on the VSPW dataset have not fully modeled spatiotemporal relationships. In this paper, we present our solution for the PVUW competition, where we introduce masked video consistency (MVC) based on existing models. MVC enforces the consistency between predictions of masked frames where random patches are withheld. The model needs to learn the segmentation results of the masked parts through the context of images and the relationship between preceding and succeeding frames of the video. Additionally, we employed test-time augmentation, model aggeregation and a multimodal model-based post-processing method. Our approach achieves 67.27% mIoU performance on the VSPW dataset, ranking 2nd place in the PVUW2024 challenge VSS track. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_04979 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Semantic Segmentation on VSPW Dataset through Masked Video Consistency Liang, Chen Guo, Qiang Yu, Chongkai Wu, Chengjing Liu, Ting Liu, Luoqi Computer Vision and Pattern Recognition Pixel-level Video Understanding requires effectively integrating three-dimensional data in both spatial and temporal dimensions to learn accurate and stable semantic information from continuous frames. However, existing advanced models on the VSPW dataset have not fully modeled spatiotemporal relationships. In this paper, we present our solution for the PVUW competition, where we introduce masked video consistency (MVC) based on existing models. MVC enforces the consistency between predictions of masked frames where random patches are withheld. The model needs to learn the segmentation results of the masked parts through the context of images and the relationship between preceding and succeeding frames of the video. Additionally, we employed test-time augmentation, model aggeregation and a multimodal model-based post-processing method. Our approach achieves 67.27% mIoU performance on the VSPW dataset, ranking 2nd place in the PVUW2024 challenge VSS track. |
| title | Semantic Segmentation on VSPW Dataset through Masked Video Consistency |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.04979 |