GroPrompt: Efficient Grounded Prompting and Adaptation for Referring Video Object Segmentation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909229910065152 |
|---|---|
| author | Lin, Ci-Siang Liu, I-Jieh Chen, Min-Hung Wang, Chien-Yi Liu, Sifei Wang, Yu-Chiang Frank |
| author_facet | Lin, Ci-Siang Liu, I-Jieh Chen, Min-Hung Wang, Chien-Yi Liu, Sifei Wang, Yu-Chiang Frank |
| contents | Referring Video Object Segmentation (RVOS) aims to segment the object referred to by the query sentence throughout the entire video. Most existing methods require end-to-end training with dense mask annotations, which could be computation-consuming and less scalable. In this work, we aim to efficiently adapt foundation segmentation models for addressing RVOS from weak supervision with the proposed Grounded Prompting (GroPrompt) framework. More specifically, we propose Text-Aware Prompt Contrastive Learning (TAP-CL) to enhance the association between the position prompts and the referring sentences with only box supervisions, including Text-Contrastive Prompt Learning (TextCon) and Modality-Contrastive Prompt Learning (ModalCon) at frame level and video level, respectively. With the proposed TAP-CL, our GroPrompt framework can generate temporal-consistent yet text-aware position prompts describing locations and movements for the referred object from the video. The experimental results in the standard RVOS benchmarks (Ref-YouTube-VOS, Ref-DAVIS17, A2D-Sentences, and JHMDB-Sentences) demonstrate the competitive performance of our proposed GroPrompt framework given only bounding box weak supervisions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_12834 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | GroPrompt: Efficient Grounded Prompting and Adaptation for Referring Video Object Segmentation Lin, Ci-Siang Liu, I-Jieh Chen, Min-Hung Wang, Chien-Yi Liu, Sifei Wang, Yu-Chiang Frank Computer Vision and Pattern Recognition Referring Video Object Segmentation (RVOS) aims to segment the object referred to by the query sentence throughout the entire video. Most existing methods require end-to-end training with dense mask annotations, which could be computation-consuming and less scalable. In this work, we aim to efficiently adapt foundation segmentation models for addressing RVOS from weak supervision with the proposed Grounded Prompting (GroPrompt) framework. More specifically, we propose Text-Aware Prompt Contrastive Learning (TAP-CL) to enhance the association between the position prompts and the referring sentences with only box supervisions, including Text-Contrastive Prompt Learning (TextCon) and Modality-Contrastive Prompt Learning (ModalCon) at frame level and video level, respectively. With the proposed TAP-CL, our GroPrompt framework can generate temporal-consistent yet text-aware position prompts describing locations and movements for the referred object from the video. The experimental results in the standard RVOS benchmarks (Ref-YouTube-VOS, Ref-DAVIS17, A2D-Sentences, and JHMDB-Sentences) demonstrate the competitive performance of our proposed GroPrompt framework given only bounding box weak supervisions. |
| title | GroPrompt: Efficient Grounded Prompting and Adaptation for Referring Video Object Segmentation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.12834 |