1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913560741806080 |
|---|---|
| author | Wang, Jinheng Zhou, Hansong Song, Ting Mao, Shaoguang Ma, Shuming Wang, Hongyu Xia, Yan Wei, Furu |
| author_facet | Wang, Jinheng Zhou, Hansong Song, Ting Mao, Shaoguang Ma, Shuming Wang, Hongyu Xia, Yan Wei, Furu |
| contents | Recent advances in 1-bit Large Language Models (LLMs), such as BitNet and BitNet b1.58, present a promising approach to enhancing the efficiency of LLMs in terms of speed and energy consumption. These developments also enable local LLM deployment across a broad range of devices. In this work, we introduce bitnet.cpp, a tailored software stack designed to unlock the full potential of 1-bit LLMs. Specifically, we develop a set of kernels to support fast and lossless inference of ternary BitNet b1.58 LLMs on CPUs. Extensive experiments demonstrate that bitnet.cpp achieves significant speedups, ranging from 2.37x to 6.17x on x86 CPUs and from 1.37x to 5.07x on ARM CPUs, across various model sizes. The code is available at https://github.com/microsoft/BitNet. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_16144 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | 1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs Wang, Jinheng Zhou, Hansong Song, Ting Mao, Shaoguang Ma, Shuming Wang, Hongyu Xia, Yan Wei, Furu Computation and Language Recent advances in 1-bit Large Language Models (LLMs), such as BitNet and BitNet b1.58, present a promising approach to enhancing the efficiency of LLMs in terms of speed and energy consumption. These developments also enable local LLM deployment across a broad range of devices. In this work, we introduce bitnet.cpp, a tailored software stack designed to unlock the full potential of 1-bit LLMs. Specifically, we develop a set of kernels to support fast and lossless inference of ternary BitNet b1.58 LLMs on CPUs. Extensive experiments demonstrate that bitnet.cpp achieves significant speedups, ranging from 2.37x to 6.17x on x86 CPUs and from 1.37x to 5.07x on ARM CPUs, across various model sizes. The code is available at https://github.com/microsoft/BitNet. |
| title | 1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.16144 |