Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.25955 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918369401241600 |
|---|---|
| author | Yang, Xiaoyu Yang, Yifan Jin, Zengrui Cui, Ziyun Wu, Wen Li, Baoxiang Zhang, Chao Woodland, Phil |
| author_facet | Yang, Xiaoyu Yang, Yifan Jin, Zengrui Cui, Ziyun Wu, Wen Li, Baoxiang Zhang, Chao Woodland, Phil |
| contents | Self-supervised learning (SSL) has significantly advanced acoustic representation learning. However, most existing models are optimised for either speech or audio event understanding, resulting in a persistent gap between these two domains. We address this gap with SPEAR (SPEech and Audio Representations), a self-supervised framework that distils complementary knowledge from a speech-focused SSL teacher and a general-audio SSL teacher into a single unified model. SPEAR applies multi-codebook vector quantisation to continuous teacher representations to produce fine-grained discrete tokens that capture both semantic and acoustic information. To effectively integrate these heterogeneous representations, SPEAR jointly predicts them given a masked input with an asymmetric pre-training loss. We further improve robustness in complex sound scenes through a novel token mixing mechanism. Extensive experiments demonstrate that SPEAR consistently outperforms existing unified speech and audio models. SPEAR establishes a new state-of-the-art on the SUPERB benchmark, surpassing WavLM Large on 12 of 15 tasks, while achieving competitive performance on the HEAR benchmark. These results position SPEAR as a versatile foundation for general-purpose speech and audio representation learning. The code and pre-trained models will be released. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_25955 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SPEAR: A Unified SSL Framework for Learning Speech and Audio Representations Yang, Xiaoyu Yang, Yifan Jin, Zengrui Cui, Ziyun Wu, Wen Li, Baoxiang Zhang, Chao Woodland, Phil Audio and Speech Processing Self-supervised learning (SSL) has significantly advanced acoustic representation learning. However, most existing models are optimised for either speech or audio event understanding, resulting in a persistent gap between these two domains. We address this gap with SPEAR (SPEech and Audio Representations), a self-supervised framework that distils complementary knowledge from a speech-focused SSL teacher and a general-audio SSL teacher into a single unified model. SPEAR applies multi-codebook vector quantisation to continuous teacher representations to produce fine-grained discrete tokens that capture both semantic and acoustic information. To effectively integrate these heterogeneous representations, SPEAR jointly predicts them given a masked input with an asymmetric pre-training loss. We further improve robustness in complex sound scenes through a novel token mixing mechanism. Extensive experiments demonstrate that SPEAR consistently outperforms existing unified speech and audio models. SPEAR establishes a new state-of-the-art on the SUPERB benchmark, surpassing WavLM Large on 12 of 15 tasks, while achieving competitive performance on the HEAR benchmark. These results position SPEAR as a versatile foundation for general-purpose speech and audio representation learning. The code and pre-trained models will be released. |
| title | SPEAR: A Unified SSL Framework for Learning Speech and Audio Representations |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2510.25955 |