Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2502.05837 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912225797603328 |
|---|---|
| author | C, Shiva Kumar Dhiman, Jitendra Kumar Adiga, Nagaraj Singh, Shatrughan |
| author_facet | C, Shiva Kumar Dhiman, Jitendra Kumar Adiga, Nagaraj Singh, Shatrughan |
| contents | Traditionally, Knowledge Distillation (KD) is used for model compression, often leading to suboptimal performance. In this paper, we evaluate the impact of combining KD loss with alternative pruning techniques, including Low-Rank Factorization (LRF) and l0 regularization, on a conformer-based pre-trained network under the paradigm of Self-Supervised Learning (SSL). We also propose a strategy to jointly prune and train an RNN-T-based ASR model, demonstrating that this approach yields superior performance compared to pruning a pre-trained network first and then using it for ASR training. This approach led to a significant reduction in word error rate: l0 and KD combination achieves the best non-streaming performance, with a 8.9% Relative Word Error Rate (RWER) improvement over the baseline, while LRF and KD combination yields the best results for streaming ASR, improving RWER by 13.4%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_05837 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Synergistic Effects of Knowledge Distillation and Structured Pruning for Self-Supervised Speech Models C, Shiva Kumar Dhiman, Jitendra Kumar Adiga, Nagaraj Singh, Shatrughan Audio and Speech Processing Traditionally, Knowledge Distillation (KD) is used for model compression, often leading to suboptimal performance. In this paper, we evaluate the impact of combining KD loss with alternative pruning techniques, including Low-Rank Factorization (LRF) and l0 regularization, on a conformer-based pre-trained network under the paradigm of Self-Supervised Learning (SSL). We also propose a strategy to jointly prune and train an RNN-T-based ASR model, demonstrating that this approach yields superior performance compared to pruning a pre-trained network first and then using it for ASR training. This approach led to a significant reduction in word error rate: l0 and KD combination achieves the best non-streaming performance, with a 8.9% Relative Word Error Rate (RWER) improvement over the baseline, while LRF and KD combination yields the best results for streaming ASR, improving RWER by 13.4%. |
| title | Synergistic Effects of Knowledge Distillation and Structured Pruning for Self-Supervised Speech Models |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2502.05837 |