Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918082640871424 |
|---|---|
| author | Tuncay, Ludovic Labbé, Etienne Benetos, Emmanouil Pellegrini, Thomas |
| author_facet | Tuncay, Ludovic Labbé, Etienne Benetos, Emmanouil Pellegrini, Thomas |
| contents | Building on the Joint-Embedding Predictive Architecture (JEPA) paradigm, a recent self-supervised learning framework that predicts latent representations of masked regions in high-level feature spaces, we propose Audio-JEPA (Audio Joint-Embedding Predictive Architecture), tailored specifically for audio data. Audio-JEPA uses a simple Vision Transformer backbone to predict latent representations of masked spectrogram patches rather than reconstructing raw audio. We pre-train on unlabeled AudioSet clips (10s, 32kHz) with random patch masking on mel-spectrograms. We evaluate on the X-ARES suite covering speech, music, and environmental sound tasks. Although our implementation is a straightforward translation of the original model to audio, the results still show comparable performance to wav2vec 2.0 and data2vec while using less than one-fifth of their training data and with no hyper-parameter tuning. All code and pretrained checkpoints will be released on GitHub. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_02915 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning Tuncay, Ludovic Labbé, Etienne Benetos, Emmanouil Pellegrini, Thomas Sound Artificial Intelligence Machine Learning Audio and Speech Processing Signal Processing Building on the Joint-Embedding Predictive Architecture (JEPA) paradigm, a recent self-supervised learning framework that predicts latent representations of masked regions in high-level feature spaces, we propose Audio-JEPA (Audio Joint-Embedding Predictive Architecture), tailored specifically for audio data. Audio-JEPA uses a simple Vision Transformer backbone to predict latent representations of masked spectrogram patches rather than reconstructing raw audio. We pre-train on unlabeled AudioSet clips (10s, 32kHz) with random patch masking on mel-spectrograms. We evaluate on the X-ARES suite covering speech, music, and environmental sound tasks. Although our implementation is a straightforward translation of the original model to audio, the results still show comparable performance to wav2vec 2.0 and data2vec while using less than one-fifth of their training data and with no hyper-parameter tuning. All code and pretrained checkpoints will be released on GitHub. |
| title | Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning |
| topic | Sound Artificial Intelligence Machine Learning Audio and Speech Processing Signal Processing |
| url | https://arxiv.org/abs/2507.02915 |