SDQ: Sparse Decomposed Quantization for LLM Inference
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911927039426560 |
|---|---|
| author | Jeong, Geonhwa Tsai, Po-An Keckler, Stephen W. Krishna, Tushar |
| author_facet | Jeong, Geonhwa Tsai, Po-An Keckler, Stephen W. Krishna, Tushar |
| contents | Recently, large language models (LLMs) have shown surprising performance in task-specific workloads as well as general tasks with the given prompts. However, to achieve unprecedented performance, recent LLMs use billions to trillions of parameters, which hinder the wide adaptation of those models due to their extremely large compute and memory requirements. To resolve the issue, various model compression methods are being actively investigated. In this work, we propose SDQ (Sparse Decomposed Quantization) to exploit both structured sparsity and quantization to achieve both high compute and memory efficiency. From our evaluations, we observe that SDQ can achieve 4x effective compute throughput with <1% quality drop. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_13868 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SDQ: Sparse Decomposed Quantization for LLM Inference Jeong, Geonhwa Tsai, Po-An Keckler, Stephen W. Krishna, Tushar Machine Learning Artificial Intelligence Recently, large language models (LLMs) have shown surprising performance in task-specific workloads as well as general tasks with the given prompts. However, to achieve unprecedented performance, recent LLMs use billions to trillions of parameters, which hinder the wide adaptation of those models due to their extremely large compute and memory requirements. To resolve the issue, various model compression methods are being actively investigated. In this work, we propose SDQ (Sparse Decomposed Quantization) to exploit both structured sparsity and quantization to achieve both high compute and memory efficiency. From our evaluations, we observe that SDQ can achieve 4x effective compute throughput with <1% quality drop. |
| title | SDQ: Sparse Decomposed Quantization for LLM Inference |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2406.13868 |