Litespark Inference on Consumer CPUs: Custom SIMD Kernels for Ternary Neural Networks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917468867395584 |
|---|---|
| author | Dade, Nii Osae Osae Morri, Tony Rahat, Moinul Hossain Pal, Sayandip |
| author_facet | Dade, Nii Osae Osae Morri, Tony Rahat, Moinul Hossain Pal, Sayandip |
| contents | Large language models (LLMs) have transformed artificial intelligence, but their computational requirements remain prohibitive for most users. Standard inference demands expensive datacenter GPUs or cloud API access, leaving over one billion personal computers underutilized for AI workloads. Ternary models offer a path forward: their weights are constrained to {-1, 0, +1}, theoretically eliminating the need for floating-point multiplication. However, existing frameworks fail to exploit this structure, treating ternary models as dense floating-point networks. We address this gap with custom SIMD kernels that replace matrix multiplication with simple addition and subtraction operations, targeting the integer dot product instructions available on modern CPUs. Our implementation, Litespark-Inference, is pip-installable and integrates directly with Hugging-Face, achieving 9.2x faster time-to-first-token, 52x higher throughput, and 14x memory reduction compared to standard PyTorch inference on Apple Silicon, with similar speedups on Intel and AMD processors. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_06485 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Litespark Inference on Consumer CPUs: Custom SIMD Kernels for Ternary Neural Networks Dade, Nii Osae Osae Morri, Tony Rahat, Moinul Hossain Pal, Sayandip Computation and Language Artificial Intelligence Large language models (LLMs) have transformed artificial intelligence, but their computational requirements remain prohibitive for most users. Standard inference demands expensive datacenter GPUs or cloud API access, leaving over one billion personal computers underutilized for AI workloads. Ternary models offer a path forward: their weights are constrained to {-1, 0, +1}, theoretically eliminating the need for floating-point multiplication. However, existing frameworks fail to exploit this structure, treating ternary models as dense floating-point networks. We address this gap with custom SIMD kernels that replace matrix multiplication with simple addition and subtraction operations, targeting the integer dot product instructions available on modern CPUs. Our implementation, Litespark-Inference, is pip-installable and integrates directly with Hugging-Face, achieving 9.2x faster time-to-first-token, 52x higher throughput, and 14x memory reduction compared to standard PyTorch inference on Apple Silicon, with similar speedups on Intel and AMD processors. |
| title | Litespark Inference on Consumer CPUs: Custom SIMD Kernels for Ternary Neural Networks |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2605.06485 |