Is Smaller Always Faster? Tradeoffs in Compressing Self-Supervised Speech Transformers
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2022
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916903096680448 |
|---|---|
| author | Lin, Tzu-Quan Yang, Tsung-Huan Chang, Chun-Yao Chen, Kuang-Ming Feng, Tzu-hsun Lee, Hung-yi Tang, Hao |
| author_facet | Lin, Tzu-Quan Yang, Tsung-Huan Chang, Chun-Yao Chen, Kuang-Ming Feng, Tzu-hsun Lee, Hung-yi Tang, Hao |
| contents | Transformer-based self-supervised models have achieved remarkable success in speech processing, but their large size and high inference cost present significant challenges for real-world deployment. While numerous compression techniques have been proposed, inconsistent evaluation metrics make it difficult to compare their practical effectiveness. In this work, we conduct a comprehensive study of four common compression methods, including weight pruning, head pruning, low-rank approximation, and knowledge distillation on self-supervised speech Transformers. We evaluate each method under three key metrics: parameter count, multiply-accumulate operations, and real-time factor. Results show that each method offers distinct advantages. In addition, we contextualize recent compression techniques, comparing DistilHuBERT, FitHuBERT, LightHuBERT, ARMHuBERT, and STaRHuBERT under the same framework, offering practical guidance on compression for deployment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2211_09949 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Is Smaller Always Faster? Tradeoffs in Compressing Self-Supervised Speech Transformers Lin, Tzu-Quan Yang, Tsung-Huan Chang, Chun-Yao Chen, Kuang-Ming Feng, Tzu-hsun Lee, Hung-yi Tang, Hao Computation and Language Machine Learning Sound Audio and Speech Processing Transformer-based self-supervised models have achieved remarkable success in speech processing, but their large size and high inference cost present significant challenges for real-world deployment. While numerous compression techniques have been proposed, inconsistent evaluation metrics make it difficult to compare their practical effectiveness. In this work, we conduct a comprehensive study of four common compression methods, including weight pruning, head pruning, low-rank approximation, and knowledge distillation on self-supervised speech Transformers. We evaluate each method under three key metrics: parameter count, multiply-accumulate operations, and real-time factor. Results show that each method offers distinct advantages. In addition, we contextualize recent compression techniques, comparing DistilHuBERT, FitHuBERT, LightHuBERT, ARMHuBERT, and STaRHuBERT under the same framework, offering practical guidance on compression for deployment. |
| title | Is Smaller Always Faster? Tradeoffs in Compressing Self-Supervised Speech Transformers |
| topic | Computation and Language Machine Learning Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2211.09949 |