Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.01192 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918042809663488 |
|---|---|
| author | Kutsakov, Aleksandr Maximenko, Alexandr Gospodinov, Georgii Bogomolov, Pavel Minkin, Fyodor |
| author_facet | Kutsakov, Aleksandr Maximenko, Alexandr Gospodinov, Georgii Bogomolov, Pavel Minkin, Fyodor |
| contents | Self-Supervised Learning (SSL) has demonstrated strong performance in speech processing, particularly in automatic speech recognition. In this paper, we explore an SSL pretraining framework that leverages masked language modeling with targets derived from a speech recognition model. We also present chunkwise attention with dynamic chunk size sampling during pretraining to enable both full-context and streaming fine-tuning. Our experiments examine scaling with respect to model size and the amount of data. Using our method, we train the GigaAM family of models, including a state-of-the-art model for Russian speech recognition that outperforms Whisper-large-v3 by 50%. We have released our foundation and ASR models, along with the inference code, under the MIT license as open-source resources to the research community. Available at https://github.com/salute-developers/gigaam. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_01192 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GigaAM: Efficient Self-Supervised Learner for Speech Recognition Kutsakov, Aleksandr Maximenko, Alexandr Gospodinov, Georgii Bogomolov, Pavel Minkin, Fyodor Audio and Speech Processing Sound Self-Supervised Learning (SSL) has demonstrated strong performance in speech processing, particularly in automatic speech recognition. In this paper, we explore an SSL pretraining framework that leverages masked language modeling with targets derived from a speech recognition model. We also present chunkwise attention with dynamic chunk size sampling during pretraining to enable both full-context and streaming fine-tuning. Our experiments examine scaling with respect to model size and the amount of data. Using our method, we train the GigaAM family of models, including a state-of-the-art model for Russian speech recognition that outperforms Whisper-large-v3 by 50%. We have released our foundation and ASR models, along with the inference code, under the MIT license as open-source resources to the research community. Available at https://github.com/salute-developers/gigaam. |
| title | GigaAM: Efficient Self-Supervised Learner for Speech Recognition |
| topic | Audio and Speech Processing Sound |
| url | https://arxiv.org/abs/2506.01192 |