An exploration of the effect of quantisation on energy consumption and inference time of StarCoder2
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917841499848704 |
|---|---|
| author | de Reus, Pepijn Oprescu, Ana Zuidema, Jelle |
| author_facet | de Reus, Pepijn Oprescu, Ana Zuidema, Jelle |
| contents | This study examines quantisation and pruning strategies to reduce energy consumption in code Large Language Models (LLMs) inference. Using StarCoder2, we observe increased energy demands with quantization due to lower throughput and some accuracy losses. Conversely, pruning reduces energy usage but impairs performance. The results highlight challenges and trade-offs in LLM model compression. We suggest future work on hardware-optimized quantization to enhance efficiency with minimal loss in accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_12758 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | An exploration of the effect of quantisation on energy consumption and inference time of StarCoder2 de Reus, Pepijn Oprescu, Ana Zuidema, Jelle Computation and Language Artificial Intelligence Software Engineering This study examines quantisation and pruning strategies to reduce energy consumption in code Large Language Models (LLMs) inference. Using StarCoder2, we observe increased energy demands with quantization due to lower throughput and some accuracy losses. Conversely, pruning reduces energy usage but impairs performance. The results highlight challenges and trade-offs in LLM model compression. We suggest future work on hardware-optimized quantization to enhance efficiency with minimal loss in accuracy. |
| title | An exploration of the effect of quantisation on energy consumption and inference time of StarCoder2 |
| topic | Computation and Language Artificial Intelligence Software Engineering |
| url | https://arxiv.org/abs/2411.12758 |