Differentiable K-means for Fully-optimized Discrete Token-based ASR
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915297871527936 |
|---|---|
| author | Onda, Kentaro Kashiwagi, Yosuke Tsunoo, Emiru Futami, Hayato Watanabe, Shinji |
| author_facet | Onda, Kentaro Kashiwagi, Yosuke Tsunoo, Emiru Futami, Hayato Watanabe, Shinji |
| contents | Recent studies have highlighted the potential of discrete tokens derived from self-supervised learning (SSL) models for various speech-related tasks. These tokens serve not only as substitutes for text in language modeling but also as intermediate representations for tasks such as automatic speech recognition (ASR). However, discrete tokens are typically obtained via k-means clustering of SSL features independently of downstream tasks, making them suboptimal for specific applications. This paper proposes the use of differentiable k-means, enabling the joint optimization of tokenization and downstream tasks. This approach enables the fine-tuning of the SSL parameters and learning weights for outputs from multiple SSL layers. Experiments were conducted with ASR as a downstream task. ASR accuracy successfully improved owing to the optimized tokens. The acquired tokens also exhibited greater purity of phonetic information, which were found to be useful even in speech resynthesis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_16207 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Differentiable K-means for Fully-optimized Discrete Token-based ASR Onda, Kentaro Kashiwagi, Yosuke Tsunoo, Emiru Futami, Hayato Watanabe, Shinji Sound Audio and Speech Processing Recent studies have highlighted the potential of discrete tokens derived from self-supervised learning (SSL) models for various speech-related tasks. These tokens serve not only as substitutes for text in language modeling but also as intermediate representations for tasks such as automatic speech recognition (ASR). However, discrete tokens are typically obtained via k-means clustering of SSL features independently of downstream tasks, making them suboptimal for specific applications. This paper proposes the use of differentiable k-means, enabling the joint optimization of tokenization and downstream tasks. This approach enables the fine-tuning of the SSL parameters and learning weights for outputs from multiple SSL layers. Experiments were conducted with ASR as a downstream task. ASR accuracy successfully improved owing to the optimized tokens. The acquired tokens also exhibited greater purity of phonetic information, which were found to be useful even in speech resynthesis. |
| title | Differentiable K-means for Fully-optimized Discrete Token-based ASR |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2505.16207 |