On Class Separability Pitfalls In Audio-Text Contrastive Zero-Shot Learning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909807316828160 |
|---|---|
| author | Tavares, Tiago Ayres, Fabio Wang, Zhepei Smaragdis, Paris |
| author_facet | Tavares, Tiago Ayres, Fabio Wang, Zhepei Smaragdis, Paris |
| contents | Recent advances in audio-text cross-modal contrastive learning have shown its potential towards zero-shot learning. One possibility for this is by projecting item embeddings from pre-trained backbone neural networks into a cross-modal space in which item similarity can be calculated in either domain. This process relies on a strong unimodal pre-training of the backbone networks, and on a data-intensive training task for the projectors. These two processes can be biased by unintentional data leakage, which can arise from using supervised learning in pre-training or from inadvertently training the cross-modal projection using labels from the zero-shot learning evaluation. In this study, we show that a significant part of the measured zero-shot learning accuracy is due to strengths inherited from the audio and text backbones, that is, they are not learned in the cross-modal domain and are not transferred from one modality to another. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_13068 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On Class Separability Pitfalls In Audio-Text Contrastive Zero-Shot Learning Tavares, Tiago Ayres, Fabio Wang, Zhepei Smaragdis, Paris Sound Machine Learning Audio and Speech Processing Recent advances in audio-text cross-modal contrastive learning have shown its potential towards zero-shot learning. One possibility for this is by projecting item embeddings from pre-trained backbone neural networks into a cross-modal space in which item similarity can be calculated in either domain. This process relies on a strong unimodal pre-training of the backbone networks, and on a data-intensive training task for the projectors. These two processes can be biased by unintentional data leakage, which can arise from using supervised learning in pre-training or from inadvertently training the cross-modal projection using labels from the zero-shot learning evaluation. In this study, we show that a significant part of the measured zero-shot learning accuracy is due to strengths inherited from the audio and text backbones, that is, they are not learned in the cross-modal domain and are not transferred from one modality to another. |
| title | On Class Separability Pitfalls In Audio-Text Contrastive Zero-Shot Learning |
| topic | Sound Machine Learning Audio and Speech Processing |
| url | https://arxiv.org/abs/2408.13068 |