Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2306.01076 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913521400283136 |
|---|---|
| author | Yang, Zi Choudhary, Samridhi Kunzmann, Siegfried Zhang, Zheng |
| author_facet | Yang, Zi Choudhary, Samridhi Kunzmann, Siegfried Zhang, Zheng |
| contents | Fine-tuned transformer models have shown superior performances in many natural language tasks. However, the large model size prohibits deploying high-performance transformer models on resource-constrained devices. This paper proposes a quantization-aware tensor-compressed training approach to reduce the model size, arithmetic operations, and ultimately runtime latency of transformer-based models. We compress the embedding and linear layers of transformers into small low-rank tensor cores, which significantly reduces model parameters. A quantization-aware training with learnable scale factors is used to further obtain low-precision representations of the tensor-compressed models. The developed approach can be used for both end-to-end training and distillation-based training. To improve the convergence, a layer-by-layer distillation is applied to distill a quantized and tensor-compressed student model from a pre-trained transformer. The performance is demonstrated in two natural language understanding tasks, showing up to $63\times$ compression ratio, little accuracy loss and remarkable inference and training speedup. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2306_01076 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Quantization-Aware and Tensor-Compressed Training of Transformers for Natural Language Understanding Yang, Zi Choudhary, Samridhi Kunzmann, Siegfried Zhang, Zheng Computation and Language Artificial Intelligence Fine-tuned transformer models have shown superior performances in many natural language tasks. However, the large model size prohibits deploying high-performance transformer models on resource-constrained devices. This paper proposes a quantization-aware tensor-compressed training approach to reduce the model size, arithmetic operations, and ultimately runtime latency of transformer-based models. We compress the embedding and linear layers of transformers into small low-rank tensor cores, which significantly reduces model parameters. A quantization-aware training with learnable scale factors is used to further obtain low-precision representations of the tensor-compressed models. The developed approach can be used for both end-to-end training and distillation-based training. To improve the convergence, a layer-by-layer distillation is applied to distill a quantized and tensor-compressed student model from a pre-trained transformer. The performance is demonstrated in two natural language understanding tasks, showing up to $63\times$ compression ratio, little accuracy loss and remarkable inference and training speedup. |
| title | Quantization-Aware and Tensor-Compressed Training of Transformers for Natural Language Understanding |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2306.01076 |