Analyzing the Importance of Blank for CTC-Based Knowledge Distillation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909633200783360 |
|---|---|
| author | Hilmes, Benedikt Rossenbach, Nick Schlüter, Ralf |
| author_facet | Hilmes, Benedikt Rossenbach, Nick Schlüter, Ralf |
| contents | With the rise of large pre-trained foundation models for automatic speech recognition new challenges appear. While the performance of these models is good, runtime and cost of inference increases. One approach to make use of their strength while retaining efficiency is to distill their knowledge to smaller models during training. In this work, we explore different CTC-based distillation variants, focusing on blank token handling. We show that common approaches like blank elimination do not always work off the shelf. We explore new blank selection patterns as a potential sweet spot between standard knowledge distillation and blank elimination mechanisms. Through the introduction of a symmetric selection method, we are able to remove the CTC loss during knowledge distillation with minimal to no performance degradation. With this, we make the training independent from target labels, potentially allowing for distillation on untranscribed audio data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_01503 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Analyzing the Importance of Blank for CTC-Based Knowledge Distillation Hilmes, Benedikt Rossenbach, Nick Schlüter, Ralf Machine Learning Sound Audio and Speech Processing With the rise of large pre-trained foundation models for automatic speech recognition new challenges appear. While the performance of these models is good, runtime and cost of inference increases. One approach to make use of their strength while retaining efficiency is to distill their knowledge to smaller models during training. In this work, we explore different CTC-based distillation variants, focusing on blank token handling. We show that common approaches like blank elimination do not always work off the shelf. We explore new blank selection patterns as a potential sweet spot between standard knowledge distillation and blank elimination mechanisms. Through the introduction of a symmetric selection method, we are able to remove the CTC loss during knowledge distillation with minimal to no performance degradation. With this, we make the training independent from target labels, potentially allowing for distillation on untranscribed audio data. |
| title | Analyzing the Importance of Blank for CTC-Based Knowledge Distillation |
| topic | Machine Learning Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2506.01503 |