Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2311.02772 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929237581103104 |
|---|---|
| author | Jeon, Sungho Yeh, Ching-Feng Inan, Hakan Hsu, Wei-Ning Rungta, Rashi Mehdad, Yashar Bikel, Daniel |
| author_facet | Jeon, Sungho Yeh, Ching-Feng Inan, Hakan Hsu, Wei-Ning Rungta, Rashi Mehdad, Yashar Bikel, Daniel |
| contents | In this paper, we show that a simple self-supervised pre-trained audio model can achieve comparable inference efficiency to more complicated pre-trained models with speech transformer encoders. These speech transformers rely on mixing convolutional modules with self-attention modules. They achieve state-of-the-art performance on ASR with top efficiency. We first show that employing these speech transformers as an encoder significantly improves the efficiency of pre-trained audio models as well. However, our study shows that we can achieve comparable efficiency with advanced self-attention solely. We demonstrate that this simpler approach is particularly beneficial with a low-bit weight quantization technique of a neural network to improve efficiency. We hypothesize that it prevents propagating the errors between different quantized modules compared to recent speech transformers mixing quantized convolution and the quantized self-attention modules. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_02772 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Attention or Convolution: Transformer Encoders in Audio Language Models for Inference Efficiency Jeon, Sungho Yeh, Ching-Feng Inan, Hakan Hsu, Wei-Ning Rungta, Rashi Mehdad, Yashar Bikel, Daniel Sound Computation and Language Audio and Speech Processing In this paper, we show that a simple self-supervised pre-trained audio model can achieve comparable inference efficiency to more complicated pre-trained models with speech transformer encoders. These speech transformers rely on mixing convolutional modules with self-attention modules. They achieve state-of-the-art performance on ASR with top efficiency. We first show that employing these speech transformers as an encoder significantly improves the efficiency of pre-trained audio models as well. However, our study shows that we can achieve comparable efficiency with advanced self-attention solely. We demonstrate that this simpler approach is particularly beneficial with a low-bit weight quantization technique of a neural network to improve efficiency. We hypothesize that it prevents propagating the errors between different quantized modules compared to recent speech transformers mixing quantized convolution and the quantized self-attention modules. |
| title | Attention or Convolution: Transformer Encoders in Audio Language Models for Inference Efficiency |
| topic | Sound Computation and Language Audio and Speech Processing |
| url | https://arxiv.org/abs/2311.02772 |