AuralSAM2: Enabling SAM2 Hear Through Pyramid Audio-Visual Feature Prompting
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909042546311168 |
|---|---|
| author | Liu, Yuyuan Chen, Yuanhong Wang, Chong Han, Junlin Wu, Junde Peng, Can Chen, Jingkun Tian, Yu Carneiro, Gustavo |
| author_facet | Liu, Yuyuan Chen, Yuanhong Wang, Chong Han, Junlin Wu, Junde Peng, Can Chen, Jingkun Tian, Yu Carneiro, Gustavo |
| contents | Segment Anything Model 2 (SAM2) exhibits strong generalisation for promptable segmentation in video clips; however, its integration with the audio modality remains underexplored. Existing approaches either convert audio into visual prompts (e.g., boxes) via foundation models, or inject adapters into the image encoder for audio-visual fusion. Yet both directions fall short in human-in-the-loop scenarios due to limited prompt accuracy and increased inference overhead. In particular, these adapter-based methods often suffer from audio prompt dilution, where the signal gradually weakens as it propagates through the network. In this work, we propose AuralSAM2, which integrates audio into SAM2 while largely preserving its promptable segmentation capability. Its core module, AuralFuser, fuses audio and visual features to generate sparse and dense prompts. Guided by audio and built upon SAM2's feature pyramid, these prompts propagate auditory cues across visual layers, reinforcing cross-modal influence. To further align modalities, we introduce an audio-guided contrastive loss that emphasises auditory relevance in dominant visual features. Our method achieves notable accuracy gains on public benchmarks with only minimal impact on the interactive efficiency of promptable segmentation. Our code is available at https://github.com/yyliu01/AuralSAM2. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_01015 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AuralSAM2: Enabling SAM2 Hear Through Pyramid Audio-Visual Feature Prompting Liu, Yuyuan Chen, Yuanhong Wang, Chong Han, Junlin Wu, Junde Peng, Can Chen, Jingkun Tian, Yu Carneiro, Gustavo Computer Vision and Pattern Recognition Segment Anything Model 2 (SAM2) exhibits strong generalisation for promptable segmentation in video clips; however, its integration with the audio modality remains underexplored. Existing approaches either convert audio into visual prompts (e.g., boxes) via foundation models, or inject adapters into the image encoder for audio-visual fusion. Yet both directions fall short in human-in-the-loop scenarios due to limited prompt accuracy and increased inference overhead. In particular, these adapter-based methods often suffer from audio prompt dilution, where the signal gradually weakens as it propagates through the network. In this work, we propose AuralSAM2, which integrates audio into SAM2 while largely preserving its promptable segmentation capability. Its core module, AuralFuser, fuses audio and visual features to generate sparse and dense prompts. Guided by audio and built upon SAM2's feature pyramid, these prompts propagate auditory cues across visual layers, reinforcing cross-modal influence. To further align modalities, we introduce an audio-guided contrastive loss that emphasises auditory relevance in dominant visual features. Our method achieves notable accuracy gains on public benchmarks with only minimal impact on the interactive efficiency of promptable segmentation. Our code is available at https://github.com/yyliu01/AuralSAM2. |
| title | AuralSAM2: Enabling SAM2 Hear Through Pyramid Audio-Visual Feature Prompting |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2506.01015 |