Impact of Frame Rates on Speech Tokenizer: A Case Study on Mandarin and English
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909648197517312 |
|---|---|
| author | Zhang, Haoyang Liu, Hexin Zhang, Xiangyu Zhang, Qiquan Hu, Yuchen Zhao, Junqi Tian, Fei Yang, Xuerui Garcia, Leibny Paola Chng, Eng Siong |
| author_facet | Zhang, Haoyang Liu, Hexin Zhang, Xiangyu Zhang, Qiquan Hu, Yuchen Zhao, Junqi Tian, Fei Yang, Xuerui Garcia, Leibny Paola Chng, Eng Siong |
| contents | The speech tokenizer plays a crucial role in recent speech tasks, generally serving as a bridge between speech signals and language models. While low-frame-rate codecs are widely employed as speech tokenizers, the impact of frame rates on speech tokens remains underexplored. In this study, we investigate how varying frame rates affect speech tokenization by examining Mandarin and English, two typologically distinct languages. We encode speech at different frame rates and evaluate the resulting semantic tokens in the speech recognition task. Our findings reveal that frame rate variations influence speech tokenization differently for each language, highlighting the interplay between frame rates, phonetic density, and language-specific acoustic features. The results provide insights into optimizing frame rate selection for speech tokenizers, with implications for automatic speech recognition, text-to-speech, and other speech-related applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_17076 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Impact of Frame Rates on Speech Tokenizer: A Case Study on Mandarin and English Zhang, Haoyang Liu, Hexin Zhang, Xiangyu Zhang, Qiquan Hu, Yuchen Zhao, Junqi Tian, Fei Yang, Xuerui Garcia, Leibny Paola Chng, Eng Siong Computation and Language Artificial Intelligence Sound Audio and Speech Processing 68T10 I.2.7 The speech tokenizer plays a crucial role in recent speech tasks, generally serving as a bridge between speech signals and language models. While low-frame-rate codecs are widely employed as speech tokenizers, the impact of frame rates on speech tokens remains underexplored. In this study, we investigate how varying frame rates affect speech tokenization by examining Mandarin and English, two typologically distinct languages. We encode speech at different frame rates and evaluate the resulting semantic tokens in the speech recognition task. Our findings reveal that frame rate variations influence speech tokenization differently for each language, highlighting the interplay between frame rates, phonetic density, and language-specific acoustic features. The results provide insights into optimizing frame rate selection for speech tokenizers, with implications for automatic speech recognition, text-to-speech, and other speech-related applications. |
| title | Impact of Frame Rates on Speech Tokenizer: A Case Study on Mandarin and English |
| topic | Computation and Language Artificial Intelligence Sound Audio and Speech Processing 68T10 I.2.7 |
| url | https://arxiv.org/abs/2505.17076 |