Enhancing Inference Efficiency of Large Language Models: Investigating Optimization Strategies and Architectural Innovations
Fuente:
arXiv
Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929306275414016 |
|---|---|
| author | Tyukin, Georgy |
| author_facet | Tyukin, Georgy |
| contents | Large Language Models are growing in size, and we expect them to continue to do so, as larger models train quicker. However, this increase in size will severely impact inference costs. Therefore model compression is important, to retain the performance of larger models, but with a reduced cost of running them. In this thesis we explore the methods of model compression, and we empirically demonstrate that the simple method of skipping latter attention sublayers in Transformer LLMs is an effective method of model compression, as these layers prove to be redundant, whilst also being incredibly computationally expensive. We observed a 21% speed increase in one-token generation for Llama 2 7B, whilst surprisingly and unexpectedly improving performance over several common benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_05741 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhancing Inference Efficiency of Large Language Models: Investigating Optimization Strategies and Architectural Innovations Tyukin, Georgy Machine Learning Artificial Intelligence Computation and Language Performance Large Language Models are growing in size, and we expect them to continue to do so, as larger models train quicker. However, this increase in size will severely impact inference costs. Therefore model compression is important, to retain the performance of larger models, but with a reduced cost of running them. In this thesis we explore the methods of model compression, and we empirically demonstrate that the simple method of skipping latter attention sublayers in Transformer LLMs is an effective method of model compression, as these layers prove to be redundant, whilst also being incredibly computationally expensive. We observed a 21% speed increase in one-token generation for Llama 2 7B, whilst surprisingly and unexpectedly improving performance over several common benchmarks. |
| title | Enhancing Inference Efficiency of Large Language Models: Investigating Optimization Strategies and Architectural Innovations |
| topic | Machine Learning Artificial Intelligence Computation and Language Performance |
| url | https://arxiv.org/abs/2404.05741 |