tinyCLAP: Distilling Constrastive Language-Audio Pretrained Models
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916407759863808 |
|---|---|
| author | Paissan, Francesco Farella, Elisabetta |
| author_facet | Paissan, Francesco Farella, Elisabetta |
| contents | Contrastive Language-Audio Pretraining (CLAP) became of crucial importance in the field of audio and speech processing. Its employment ranges from sound event detection to text-to-audio generation. However, one of the main limitations is the considerable amount of data required in the training process and the overall computational complexity during inference. This paper investigates how we can reduce the complexity of contrastive language-audio pre-trained models, yielding an efficient model that we call tinyCLAP. We derive an unimodal distillation loss from first principles and explore how the dimensionality of the shared, multimodal latent space can be reduced via pruning. TinyCLAP uses only 6% of the original Microsoft CLAP parameters with a minimal reduction (less than 5%) in zero-shot classification performance across the three sound event detection datasets on which it was tested |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_14517 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | tinyCLAP: Distilling Constrastive Language-Audio Pretrained Models Paissan, Francesco Farella, Elisabetta Sound Computation and Language Machine Learning Audio and Speech Processing Contrastive Language-Audio Pretraining (CLAP) became of crucial importance in the field of audio and speech processing. Its employment ranges from sound event detection to text-to-audio generation. However, one of the main limitations is the considerable amount of data required in the training process and the overall computational complexity during inference. This paper investigates how we can reduce the complexity of contrastive language-audio pre-trained models, yielding an efficient model that we call tinyCLAP. We derive an unimodal distillation loss from first principles and explore how the dimensionality of the shared, multimodal latent space can be reduced via pruning. TinyCLAP uses only 6% of the original Microsoft CLAP parameters with a minimal reduction (less than 5%) in zero-shot classification performance across the three sound event detection datasets on which it was tested |
| title | tinyCLAP: Distilling Constrastive Language-Audio Pretrained Models |
| topic | Sound Computation and Language Machine Learning Audio and Speech Processing |
| url | https://arxiv.org/abs/2311.14517 |