Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.08680 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909314602500096 |
|---|---|
| author | Han, Minglun Bai, Ye Shen, Chen Huang, Youjia Huang, Mingkun Lin, Zehua Dong, Linhao Lu, Lu Wang, Yuxuan |
| author_facet | Han, Minglun Bai, Ye Shen, Chen Huang, Youjia Huang, Mingkun Lin, Zehua Dong, Linhao Lu, Lu Wang, Yuxuan |
| contents | Speech self-supervised pre-training can effectively improve the performance of downstream tasks. However, previous self-supervised learning (SSL) methods for speech, such as HuBERT and BEST-RQ, focus on utilizing non-causal encoders with bidirectional context, and lack sufficient support for downstream streaming models. To address this issue, we introduce the next token prediction based speech pre-training method with random-projection quantizer (NEST-RQ). NEST-RQ employs causal encoders with only left context and uses next token prediction (NTP) as the training task. On the large-scale dataset, compared to BEST-RQ, the proposed NEST-RQ achieves comparable performance on non-streaming automatic speech recognition (ASR) and better performance on streaming ASR. We also conduct analytical experiments in terms of the future context size of streaming ASR, the codebook quality of SSL and the model size of the encoder. In summary, the paper demonstrates the feasibility of the NTP in speech SSL and provides empirical evidence and insights for speech SSL research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_08680 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | NEST-RQ: Next Token Prediction for Speech Self-Supervised Pre-Training Han, Minglun Bai, Ye Shen, Chen Huang, Youjia Huang, Mingkun Lin, Zehua Dong, Linhao Lu, Lu Wang, Yuxuan Audio and Speech Processing Artificial Intelligence Computation and Language Speech self-supervised pre-training can effectively improve the performance of downstream tasks. However, previous self-supervised learning (SSL) methods for speech, such as HuBERT and BEST-RQ, focus on utilizing non-causal encoders with bidirectional context, and lack sufficient support for downstream streaming models. To address this issue, we introduce the next token prediction based speech pre-training method with random-projection quantizer (NEST-RQ). NEST-RQ employs causal encoders with only left context and uses next token prediction (NTP) as the training task. On the large-scale dataset, compared to BEST-RQ, the proposed NEST-RQ achieves comparable performance on non-streaming automatic speech recognition (ASR) and better performance on streaming ASR. We also conduct analytical experiments in terms of the future context size of streaming ASR, the codebook quality of SSL and the model size of the encoder. In summary, the paper demonstrates the feasibility of the NTP in speech SSL and provides empirical evidence and insights for speech SSL research. |
| title | NEST-RQ: Next Token Prediction for Speech Self-Supervised Pre-Training |
| topic | Audio and Speech Processing Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2409.08680 |