Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.26207 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918390987227136 |
|---|---|
| author | Diecidue, Andrea Barbano, Carlo Alberto Fraternali, Piero Fontaine, Mathieu Tartaglione, Enzo |
| author_facet | Diecidue, Andrea Barbano, Carlo Alberto Fraternali, Piero Fontaine, Mathieu Tartaglione, Enzo |
| contents | Transformer-based models have become the state of the art across multiple domains, from natural language processing to machine listening, thanks to the attention mechanisms. However, the attention layers require a large number of parameters and high-end hardware for both training and inference. We propose a novel channel-pruning technique explicitly targeted at the attention mechanism, decoupling the pruning of each head and the four layers in the attention block: query, key, value, and output projection matrices, employing a second-order metric to score the network's parameters. We compare our technique against head-pruning strategies and magnitude-driven scoring metrics, investigating the effects of pruning on Audio Spectrogram Transformer (AST) and Whisper. Our results show that even after pruning 50\% of the parameters in the attention block, performance is largely preserved. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_26207 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The silence of the weights: a structural pruning strategy for attention-based audio signal architectures with second order metrics Diecidue, Andrea Barbano, Carlo Alberto Fraternali, Piero Fontaine, Mathieu Tartaglione, Enzo Sound Machine Learning Transformer-based models have become the state of the art across multiple domains, from natural language processing to machine listening, thanks to the attention mechanisms. However, the attention layers require a large number of parameters and high-end hardware for both training and inference. We propose a novel channel-pruning technique explicitly targeted at the attention mechanism, decoupling the pruning of each head and the four layers in the attention block: query, key, value, and output projection matrices, employing a second-order metric to score the network's parameters. We compare our technique against head-pruning strategies and magnitude-driven scoring metrics, investigating the effects of pruning on Audio Spectrogram Transformer (AST) and Whisper. Our results show that even after pruning 50\% of the parameters in the attention block, performance is largely preserved. |
| title | The silence of the weights: a structural pruning strategy for attention-based audio signal architectures with second order metrics |
| topic | Sound Machine Learning |
| url | https://arxiv.org/abs/2509.26207 |