Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2312.09211 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929209662767104 |
|---|---|
| author | Ghaffari, Alireza Yu, Justin Nejad, Mahsa Ghazvini Asgharian, Masoud Chen, Boxing Nia, Vahid Partovi |
| author_facet | Ghaffari, Alireza Yu, Justin Nejad, Mahsa Ghazvini Asgharian, Masoud Chen, Boxing Nia, Vahid Partovi |
| contents | Low-precision fine-tuning of language models has gained prominence as a cost-effective and energy-efficient approach to deploying large-scale models in various applications. However, this approach is susceptible to the existence of outlier values in activation. The outlier values in the activation can negatively affect the performance of fine-tuning language models in the low-precision regime since they affect the scaling factor and thus make representing smaller values harder. This paper investigates techniques for mitigating outlier activation in low-precision integer fine-tuning of the language models. Our proposed novel approach enables us to represent the outlier activation values in 8-bit integers instead of floating-point (FP16) values. The benefit of using integers for outlier values is that it enables us to use operator tiling to avoid performing 16-bit integer matrix multiplication to address this problem effectively. We provide theoretical analysis and supporting experiments to demonstrate the effectiveness of our approach in improving the robustness and performance of low-precision fine-tuned language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_09211 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Mitigating Outlier Activations in Low-Precision Fine-Tuning of Language Models Ghaffari, Alireza Yu, Justin Nejad, Mahsa Ghazvini Asgharian, Masoud Chen, Boxing Nia, Vahid Partovi Computation and Language Low-precision fine-tuning of language models has gained prominence as a cost-effective and energy-efficient approach to deploying large-scale models in various applications. However, this approach is susceptible to the existence of outlier values in activation. The outlier values in the activation can negatively affect the performance of fine-tuning language models in the low-precision regime since they affect the scaling factor and thus make representing smaller values harder. This paper investigates techniques for mitigating outlier activation in low-precision integer fine-tuning of the language models. Our proposed novel approach enables us to represent the outlier activation values in 8-bit integers instead of floating-point (FP16) values. The benefit of using integers for outlier values is that it enables us to use operator tiling to avoid performing 16-bit integer matrix multiplication to address this problem effectively. We provide theoretical analysis and supporting experiments to demonstrate the effectiveness of our approach in improving the robustness and performance of low-precision fine-tuned language models. |
| title | Mitigating Outlier Activations in Low-Precision Fine-Tuning of Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2312.09211 |