Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2505.18860 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908378793508864 |
|---|---|
| author | Someki, Masao Bharadwaj, Shikhar Joshi, Atharva Anand Lin, Chyi-Jiunn Tian, Jinchuan Jung, Jee-weon Müller, Markus Susanj, Nathan Liu, Jing Watanabe, Shinji |
| author_facet | Someki, Masao Bharadwaj, Shikhar Joshi, Atharva Anand Lin, Chyi-Jiunn Tian, Jinchuan Jung, Jee-weon Müller, Markus Susanj, Nathan Liu, Jing Watanabe, Shinji |
| contents | Speech foundation models achieve strong generalization across languages and acoustic conditions, but require significant computational resources for inference. In the context of speech foundation models, pruning techniques have been studied that dynamically optimize model structures based on the target audio leveraging external context. In this work, we extend this line of research and propose context-driven dynamic pruning, a technique that optimizes the model computation depending on the context between different input frames and additional context during inference. We employ the Open Whisper-style Speech Model (OWSM) and incorporate speaker embeddings, acoustic event embeddings, and language information as additional context. By incorporating the speaker embedding, our method achieves a reduction of 56.7 GFLOPs while improving BLEU scores by a relative 25.7% compared to the fully fine-tuned OWSM model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_18860 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Context-Driven Dynamic Pruning for Large Speech Foundation Models Someki, Masao Bharadwaj, Shikhar Joshi, Atharva Anand Lin, Chyi-Jiunn Tian, Jinchuan Jung, Jee-weon Müller, Markus Susanj, Nathan Liu, Jing Watanabe, Shinji Audio and Speech Processing Speech foundation models achieve strong generalization across languages and acoustic conditions, but require significant computational resources for inference. In the context of speech foundation models, pruning techniques have been studied that dynamically optimize model structures based on the target audio leveraging external context. In this work, we extend this line of research and propose context-driven dynamic pruning, a technique that optimizes the model computation depending on the context between different input frames and additional context during inference. We employ the Open Whisper-style Speech Model (OWSM) and incorporate speaker embeddings, acoustic event embeddings, and language information as additional context. By incorporating the speaker embedding, our method achieves a reduction of 56.7 GFLOPs while improving BLEU scores by a relative 25.7% compared to the fully fine-tuned OWSM model. |
| title | Context-Driven Dynamic Pruning for Large Speech Foundation Models |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2505.18860 |