Investigating Design Choices in Joint-Embedding Predictive Architectures for General Audio Representation Learning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917665607516160 |
|---|---|
| author | Riou, Alain Lattner, Stefan Hadjeres, Gaëtan Peeters, Geoffroy |
| author_facet | Riou, Alain Lattner, Stefan Hadjeres, Gaëtan Peeters, Geoffroy |
| contents | This paper addresses the problem of self-supervised general-purpose audio representation learning. We explore the use of Joint-Embedding Predictive Architectures (JEPA) for this task, which consists of splitting an input mel-spectrogram into two parts (context and target), computing neural representations for each, and training the neural network to predict the target representations from the context representations. We investigate several design choices within this framework and study their influence through extensive experiments by evaluating our models on various audio classification benchmarks, including environmental sounds, speech and music downstream tasks. We focus notably on which part of the input data is used as context or target and show experimentally that it significantly impacts the model's quality. In particular, we notice that some effective design choices in the image domain lead to poor performance on audio, thus highlighting major differences between these two modalities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_08679 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Investigating Design Choices in Joint-Embedding Predictive Architectures for General Audio Representation Learning Riou, Alain Lattner, Stefan Hadjeres, Gaëtan Peeters, Geoffroy Sound Artificial Intelligence Machine Learning Audio and Speech Processing This paper addresses the problem of self-supervised general-purpose audio representation learning. We explore the use of Joint-Embedding Predictive Architectures (JEPA) for this task, which consists of splitting an input mel-spectrogram into two parts (context and target), computing neural representations for each, and training the neural network to predict the target representations from the context representations. We investigate several design choices within this framework and study their influence through extensive experiments by evaluating our models on various audio classification benchmarks, including environmental sounds, speech and music downstream tasks. We focus notably on which part of the input data is used as context or target and show experimentally that it significantly impacts the model's quality. In particular, we notice that some effective design choices in the image domain lead to poor performance on audio, thus highlighting major differences between these two modalities. |
| title | Investigating Design Choices in Joint-Embedding Predictive Architectures for General Audio Representation Learning |
| topic | Sound Artificial Intelligence Machine Learning Audio and Speech Processing |
| url | https://arxiv.org/abs/2405.08679 |