An Exploration of Mamba for Speech Self-Supervised Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911606638641152 |
|---|---|
| author | Lin, Tzu-Quan Kuo, Heng-Cheng Wei, Tzu-Chieh Cheng, Hsi-Chun Chen, Chun Wei Hsiao, Hsien-Fu Tsao, Yu Lee, Hung-yi |
| author_facet | Lin, Tzu-Quan Kuo, Heng-Cheng Wei, Tzu-Chieh Cheng, Hsi-Chun Chen, Chun Wei Hsiao, Hsien-Fu Tsao, Yu Lee, Hung-yi |
| contents | While Mamba has demonstrated strong performance in language modeling, its potential as a speech self-supervised learning (SSL) model remains underexplored, with prior studies limited to isolated tasks. To address this, we explore Mamba-based HuBERT models as alternatives to Transformer-based SSL architectures. Leveraging the linear-time Selective State Space, these models enable fine-tuning on long-context ASR with significantly lower compute. Moreover, they show superior performance when fine-tuned for streaming ASR. Beyond fine-tuning, these models show competitive performance on SUPERB probing benchmarks, particularly in causal settings. Our analysis shows that they yield higher-quality quantized representations and capture speaker-related features more distinctly than Transformer-based models. These findings highlight Mamba-based SSL as a promising and complementary direction for long-sequence modeling, real-time speech modeling, and speech unit extraction. The codebase is available at https://github.com/hckuo145/Mamba-based-HuBERT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_12606 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | An Exploration of Mamba for Speech Self-Supervised Models Lin, Tzu-Quan Kuo, Heng-Cheng Wei, Tzu-Chieh Cheng, Hsi-Chun Chen, Chun Wei Hsiao, Hsien-Fu Tsao, Yu Lee, Hung-yi Computation and Language Artificial Intelligence While Mamba has demonstrated strong performance in language modeling, its potential as a speech self-supervised learning (SSL) model remains underexplored, with prior studies limited to isolated tasks. To address this, we explore Mamba-based HuBERT models as alternatives to Transformer-based SSL architectures. Leveraging the linear-time Selective State Space, these models enable fine-tuning on long-context ASR with significantly lower compute. Moreover, they show superior performance when fine-tuned for streaming ASR. Beyond fine-tuning, these models show competitive performance on SUPERB probing benchmarks, particularly in causal settings. Our analysis shows that they yield higher-quality quantized representations and capture speaker-related features more distinctly than Transformer-based models. These findings highlight Mamba-based SSL as a promising and complementary direction for long-sequence modeling, real-time speech modeling, and speech unit extraction. The codebase is available at https://github.com/hckuo145/Mamba-based-HuBERT. |
| title | An Exploration of Mamba for Speech Self-Supervised Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2506.12606 |