Unfolding A Few Structures for The Many: Memory-Efficient Compression of Conformer and Speech Foundation Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916761917456384 |
|---|---|
| author | Li, Zhaoqing Xu, Haoning Xie, Xurong Jin, Zengrui Wang, Tianzi Liu, Xunying |
| author_facet | Li, Zhaoqing Xu, Haoning Xie, Xurong Jin, Zengrui Wang, Tianzi Liu, Xunying |
| contents | This paper presents a novel memory-efficient model compression approach for Conformer ASR and speech foundation systems. Our approach features a unique "small-to-large" design. A compact "seed" model containing a few Conformer or Transformer blocks is trained and unfolded many times to emulate the performance of larger uncompressed models with different logical depths. The seed model and many unfolded paths are jointly trained within a single unfolding cycle. The KL-divergence between the largest unfolded and smallest seed models is used in a self-distillation process to minimize their performance disparity. Experimental results show that our foldable model produces ASR performance comparable to individually constructed Conformer and wav2vec2/HuBERT speech foundation models under various depth configurations, while requiring only minimal memory and storage. Conformer and wav2vec2 models with a reduction of 35% and 30% parameters are obtained without loss of performance, respectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_21237 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Unfolding A Few Structures for The Many: Memory-Efficient Compression of Conformer and Speech Foundation Models Li, Zhaoqing Xu, Haoning Xie, Xurong Jin, Zengrui Wang, Tianzi Liu, Xunying Sound Audio and Speech Processing This paper presents a novel memory-efficient model compression approach for Conformer ASR and speech foundation systems. Our approach features a unique "small-to-large" design. A compact "seed" model containing a few Conformer or Transformer blocks is trained and unfolded many times to emulate the performance of larger uncompressed models with different logical depths. The seed model and many unfolded paths are jointly trained within a single unfolding cycle. The KL-divergence between the largest unfolded and smallest seed models is used in a self-distillation process to minimize their performance disparity. Experimental results show that our foldable model produces ASR performance comparable to individually constructed Conformer and wav2vec2/HuBERT speech foundation models under various depth configurations, while requiring only minimal memory and storage. Conformer and wav2vec2 models with a reduction of 35% and 30% parameters are obtained without loss of performance, respectively. |
| title | Unfolding A Few Structures for The Many: Memory-Efficient Compression of Conformer and Speech Foundation Models |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2505.21237 |