Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.11722 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913541472124928 |
|---|---|
| author | Chitsaz, Kamran Fournier, Quentin Mordido, Gonçalo Chandar, Sarath |
| author_facet | Chitsaz, Kamran Fournier, Quentin Mordido, Gonçalo Chandar, Sarath |
| contents | The increasing scale of Transformer models has led to an increase in their pre-training computational requirements. While quantization has proven to be effective after pre-training and during fine-tuning, applying quantization in Transformers during pre-training has remained largely unexplored at scale for language modeling. This study aims to explore the impact of quantization for efficient pre-training of Transformers, with a focus on linear layer components. By systematically applying straightforward linear quantization to weights, activations, gradients, and optimizer states, we assess its effects on model efficiency, stability, and performance during training. By offering a comprehensive recipe of effective quantization strategies to be applied during the pre-training of Transformers, we promote high training efficiency from scratch while retaining language modeling ability. Code is available at https://github.com/chandar-lab/EfficientLLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_11722 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Exploring Quantization for Efficient Pre-Training of Transformer Language Models Chitsaz, Kamran Fournier, Quentin Mordido, Gonçalo Chandar, Sarath Machine Learning The increasing scale of Transformer models has led to an increase in their pre-training computational requirements. While quantization has proven to be effective after pre-training and during fine-tuning, applying quantization in Transformers during pre-training has remained largely unexplored at scale for language modeling. This study aims to explore the impact of quantization for efficient pre-training of Transformers, with a focus on linear layer components. By systematically applying straightforward linear quantization to weights, activations, gradients, and optimizer states, we assess its effects on model efficiency, stability, and performance during training. By offering a comprehensive recipe of effective quantization strategies to be applied during the pre-training of Transformers, we promote high training efficiency from scratch while retaining language modeling ability. Code is available at https://github.com/chandar-lab/EfficientLLMs. |
| title | Exploring Quantization for Efficient Pre-Training of Transformer Language Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2407.11722 |