You Only Speak Once to See
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914960540434432 |
|---|---|
| author | Yang, Wenhao Wei, Jianguo Lu, Wenhuan Li, Lei |
| author_facet | Yang, Wenhao Wei, Jianguo Lu, Wenhuan Li, Lei |
| contents | Grounding objects in images using visual cues is a well-established approach in computer vision, yet the potential of audio as a modality for object recognition and grounding remains underexplored. We introduce YOSS, "You Only Speak Once to See," to leverage audio for grounding objects in visual scenes, termed Audio Grounding. By integrating pre-trained audio models with visual models using contrastive learning and multi-modal alignment, our approach captures speech commands or descriptions and maps them directly to corresponding objects within images. Experimental results indicate that audio guidance can be effectively applied to object grounding, suggesting that incorporating audio guidance may enhance the precision and robustness of current object grounding methods and improve the performance of robotic systems and computer vision applications. This finding opens new possibilities for advanced object recognition, scene understanding, and the development of more intuitive and capable robotic systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_18372 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | You Only Speak Once to See Yang, Wenhao Wei, Jianguo Lu, Wenhuan Li, Lei Computer Vision and Pattern Recognition Grounding objects in images using visual cues is a well-established approach in computer vision, yet the potential of audio as a modality for object recognition and grounding remains underexplored. We introduce YOSS, "You Only Speak Once to See," to leverage audio for grounding objects in visual scenes, termed Audio Grounding. By integrating pre-trained audio models with visual models using contrastive learning and multi-modal alignment, our approach captures speech commands or descriptions and maps them directly to corresponding objects within images. Experimental results indicate that audio guidance can be effectively applied to object grounding, suggesting that incorporating audio guidance may enhance the precision and robustness of current object grounding methods and improve the performance of robotic systems and computer vision applications. This finding opens new possibilities for advanced object recognition, scene understanding, and the development of more intuitive and capable robotic systems. |
| title | You Only Speak Once to See |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2409.18372 |