Learning What To Hear: Boosting Sound-Source Association For Robust Audiovisual Instance Segmentation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908795125366784 |
|---|---|
| author | Seo, Jinbae Kwon, Hyeongjun Kim, Kwonyoung Lee, Jiyoung Sohn, Kwanghoon |
| author_facet | Seo, Jinbae Kwon, Hyeongjun Kim, Kwonyoung Lee, Jiyoung Sohn, Kwanghoon |
| contents | Audiovisual instance segmentation (AVIS) requires accurately localizing and tracking sounding objects throughout video sequences. Existing methods suffer from visual bias stemming from two fundamental issues: uniform additive fusion prevents queries from specializing to different sound sources, while visual-only training objectives allow queries to converge to arbitrary salient objects. We propose Audio-Centric Query Generation using cross-attention, enabling each query to selectively attend to distinct sound sources and carry sound-specific priors into visual decoding. Additionally, we introduce Sound-Aware Ordinal Counting (SAOC) loss that explicitly supervises sounding object numbers through ordinal regression with monotonic consistency constraints, preventing visual-only convergence during training. Experiments on AVISeg benchmark demonstrate consistent improvements: +1.64 mAP, +0.6 HOTA, and +2.06 FSLA, validating that query specialization and explicit counting supervision are crucial for accurate audiovisual instance segmentation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_22740 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Learning What To Hear: Boosting Sound-Source Association For Robust Audiovisual Instance Segmentation Seo, Jinbae Kwon, Hyeongjun Kim, Kwonyoung Lee, Jiyoung Sohn, Kwanghoon Audio and Speech Processing Artificial Intelligence Multimedia Sound Audiovisual instance segmentation (AVIS) requires accurately localizing and tracking sounding objects throughout video sequences. Existing methods suffer from visual bias stemming from two fundamental issues: uniform additive fusion prevents queries from specializing to different sound sources, while visual-only training objectives allow queries to converge to arbitrary salient objects. We propose Audio-Centric Query Generation using cross-attention, enabling each query to selectively attend to distinct sound sources and carry sound-specific priors into visual decoding. Additionally, we introduce Sound-Aware Ordinal Counting (SAOC) loss that explicitly supervises sounding object numbers through ordinal regression with monotonic consistency constraints, preventing visual-only convergence during training. Experiments on AVISeg benchmark demonstrate consistent improvements: +1.64 mAP, +0.6 HOTA, and +2.06 FSLA, validating that query specialization and explicit counting supervision are crucial for accurate audiovisual instance segmentation. |
| title | Learning What To Hear: Boosting Sound-Source Association For Robust Audiovisual Instance Segmentation |
| topic | Audio and Speech Processing Artificial Intelligence Multimedia Sound |
| url | https://arxiv.org/abs/2509.22740 |