Optimization of Armv9 architecture general large language model inference performance based on Llama.cpp
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911919247458304 |
|---|---|
| author | Chen, Longhao Zhao, Yina Xie, Qiangjun Sheng, Qinghua |
| author_facet | Chen, Longhao Zhao, Yina Xie, Qiangjun Sheng, Qinghua |
| contents | This article optimizes the inference performance of the Qwen-1.8B model by performing Int8 quantization, vectorizing some operators in llama.cpp, and modifying the compilation script to improve the compiler optimization level. On the Yitian 710 experimental platform, the prefill performance is increased by 1.6 times, the decoding performance is increased by 24 times, the memory usage is reduced to 1/5 of the original, and the accuracy loss is almost negligible. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_10816 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Optimization of Armv9 architecture general large language model inference performance based on Llama.cpp Chen, Longhao Zhao, Yina Xie, Qiangjun Sheng, Qinghua Programming Languages Artificial Intelligence Hardware Architecture Performance This article optimizes the inference performance of the Qwen-1.8B model by performing Int8 quantization, vectorizing some operators in llama.cpp, and modifying the compilation script to improve the compiler optimization level. On the Yitian 710 experimental platform, the prefill performance is increased by 1.6 times, the decoding performance is increased by 24 times, the memory usage is reduced to 1/5 of the original, and the accuracy loss is almost negligible. |
| title | Optimization of Armv9 architecture general large language model inference performance based on Llama.cpp |
| topic | Programming Languages Artificial Intelligence Hardware Architecture Performance |
| url | https://arxiv.org/abs/2406.10816 |