WaterVG: Waterway Visual Grounding based on Text-Guided Vision and mmWave Radar
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914741917581312 |
|---|---|
| author | Guan, Runwei Jia, Liye Yang, Fengyufan Yao, Shanliang Purwanto, Erick Zhu, Xiaohui Lim, Eng Gee Smith, Jeremy Man, Ka Lok Hu, Xuming Yue, Yutao |
| author_facet | Guan, Runwei Jia, Liye Yang, Fengyufan Yao, Shanliang Purwanto, Erick Zhu, Xiaohui Lim, Eng Gee Smith, Jeremy Man, Ka Lok Hu, Xuming Yue, Yutao |
| contents | The perception of waterways based on human intent is significant for autonomous navigation and operations of Unmanned Surface Vehicles (USVs) in water environments. Inspired by visual grounding, we introduce WaterVG, the first visual grounding dataset designed for USV-based waterway perception based on human prompts. WaterVG encompasses prompts describing multiple targets, with annotations at the instance level including bounding boxes and masks. Notably, WaterVG includes 11,568 samples with 34,987 referred targets, whose prompts integrates both visual and radar characteristics. The pattern of text-guided two sensors equips a finer granularity of text prompts with visual and radar features of referred targets. Moreover, we propose a low-power visual grounding model, Potamoi, which is a multi-task model with a well-designed Phased Heterogeneous Modality Fusion (PHMF) mode, including Adaptive Radar Weighting (ARW) and Multi-Head Slim Cross Attention (MHSCA). Exactly, ARW extracts required radar features to fuse with vision for prompt alignment. MHSCA is an efficient fusion module with a remarkably small parameter count and FLOPs, elegantly fusing scenario context captured by two sensors with linguistic features, which performs expressively on visual grounding tasks. Comprehensive experiments and evaluations have been conducted on WaterVG, where our Potamoi archives state-of-the-art performances compared with counterparts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_12686 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | WaterVG: Waterway Visual Grounding based on Text-Guided Vision and mmWave Radar Guan, Runwei Jia, Liye Yang, Fengyufan Yao, Shanliang Purwanto, Erick Zhu, Xiaohui Lim, Eng Gee Smith, Jeremy Man, Ka Lok Hu, Xuming Yue, Yutao Computer Vision and Pattern Recognition Multimedia Robotics The perception of waterways based on human intent is significant for autonomous navigation and operations of Unmanned Surface Vehicles (USVs) in water environments. Inspired by visual grounding, we introduce WaterVG, the first visual grounding dataset designed for USV-based waterway perception based on human prompts. WaterVG encompasses prompts describing multiple targets, with annotations at the instance level including bounding boxes and masks. Notably, WaterVG includes 11,568 samples with 34,987 referred targets, whose prompts integrates both visual and radar characteristics. The pattern of text-guided two sensors equips a finer granularity of text prompts with visual and radar features of referred targets. Moreover, we propose a low-power visual grounding model, Potamoi, which is a multi-task model with a well-designed Phased Heterogeneous Modality Fusion (PHMF) mode, including Adaptive Radar Weighting (ARW) and Multi-Head Slim Cross Attention (MHSCA). Exactly, ARW extracts required radar features to fuse with vision for prompt alignment. MHSCA is an efficient fusion module with a remarkably small parameter count and FLOPs, elegantly fusing scenario context captured by two sensors with linguistic features, which performs expressively on visual grounding tasks. Comprehensive experiments and evaluations have been conducted on WaterVG, where our Potamoi archives state-of-the-art performances compared with counterparts. |
| title | WaterVG: Waterway Visual Grounding based on Text-Guided Vision and mmWave Radar |
| topic | Computer Vision and Pattern Recognition Multimedia Robotics |
| url | https://arxiv.org/abs/2403.12686 |