TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915304439808000 |
|---|---|
| author | Malard, Hugo Olvera, Michel Lathuiliere, Stephane Essid, Slim |
| author_facet | Malard, Hugo Olvera, Michel Lathuiliere, Stephane Essid, Slim |
| contents | Large-scale pre-trained audio and image models demonstrate an unprecedented degree of generalization, making them suitable for a wide range of applications. Here, we tackle the specific task of sound-prompted segmentation, aiming to segment image regions corresponding to objects heard in an audio signal. Most existing approaches tackle this problem by fine-tuning pre-trained models or by training additional modules specifically for the task. We adopt a different strategy: we introduce a training-free approach that leverages Non-negative Matrix Factorization (NMF) to co-factorize audio and visual features from pre-trained models so as to reveal shared interpretable concepts. These concepts are passed on to an open-vocabulary segmentation model for precise segmentation maps. By using frozen pre-trained models, our method achieves high generalization and establishes state-of-the-art performance in unsupervised sound-prompted segmentation, significantly surpassing previous unsupervised methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_01488 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization Malard, Hugo Olvera, Michel Lathuiliere, Stephane Essid, Slim Audio and Speech Processing Machine Learning Image and Video Processing Large-scale pre-trained audio and image models demonstrate an unprecedented degree of generalization, making them suitable for a wide range of applications. Here, we tackle the specific task of sound-prompted segmentation, aiming to segment image regions corresponding to objects heard in an audio signal. Most existing approaches tackle this problem by fine-tuning pre-trained models or by training additional modules specifically for the task. We adopt a different strategy: we introduce a training-free approach that leverages Non-negative Matrix Factorization (NMF) to co-factorize audio and visual features from pre-trained models so as to reveal shared interpretable concepts. These concepts are passed on to an open-vocabulary segmentation model for precise segmentation maps. By using frozen pre-trained models, our method achieves high generalization and establishes state-of-the-art performance in unsupervised sound-prompted segmentation, significantly surpassing previous unsupervised methods. |
| title | TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization |
| topic | Audio and Speech Processing Machine Learning Image and Video Processing |
| url | https://arxiv.org/abs/2412.01488 |