Inferflow: an Efficient and Highly Configurable Inference Engine for Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929210667302912 |
|---|---|
| author | Shi, Shuming Zhao, Enbo Cai, Deng Cui, Leyang Huang, Xinting Li, Huayang |
| author_facet | Shi, Shuming Zhao, Enbo Cai, Deng Cui, Leyang Huang, Xinting Li, Huayang |
| contents | We present Inferflow, an efficient and highly configurable inference engine for large language models (LLMs). With Inferflow, users can serve most of the common transformer models by simply modifying some lines in corresponding configuration files, without writing a single line of source code. Compared with most existing inference engines, Inferflow has some key features. First, by implementing a modular framework of atomic build-blocks and technologies, Inferflow is compositionally generalizable to new models. Second, 3.5-bit quantization is introduced in Inferflow as a tradeoff between 3-bit and 4-bit quantization. Third, hybrid model partitioning for multi-GPU inference is introduced in Inferflow to better balance inference speed and throughput than the existing partition-by-layer and partition-by-tensor strategies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_08294 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Inferflow: an Efficient and Highly Configurable Inference Engine for Large Language Models Shi, Shuming Zhao, Enbo Cai, Deng Cui, Leyang Huang, Xinting Li, Huayang Computation and Language We present Inferflow, an efficient and highly configurable inference engine for large language models (LLMs). With Inferflow, users can serve most of the common transformer models by simply modifying some lines in corresponding configuration files, without writing a single line of source code. Compared with most existing inference engines, Inferflow has some key features. First, by implementing a modular framework of atomic build-blocks and technologies, Inferflow is compositionally generalizable to new models. Second, 3.5-bit quantization is introduced in Inferflow as a tradeoff between 3-bit and 4-bit quantization. Third, hybrid model partitioning for multi-GPU inference is introduced in Inferflow to better balance inference speed and throughput than the existing partition-by-layer and partition-by-tensor strategies. |
| title | Inferflow: an Efficient and Highly Configurable Inference Engine for Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2401.08294 |