V-Seek: Accelerating LLM Reasoning on Open-hardware Server-class RISC-V Platforms
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913751227170816 |
|---|---|
| author | Rodrigo, Javier J. Poveda Ahmdi, Mohamed Amine Burrello, Alessio Pagliari, Daniele Jahier Benini, Luca |
| author_facet | Rodrigo, Javier J. Poveda Ahmdi, Mohamed Amine Burrello, Alessio Pagliari, Daniele Jahier Benini, Luca |
| contents | The recent exponential growth of Large Language Models (LLMs) has relied on GPU-based systems. However, CPUs are emerging as a flexible and lower-cost alternative, especially when targeting inference and reasoning workloads. RISC-V is rapidly gaining traction in this area, given its open and vendor-neutral ISA. However, the RISC-V hardware for LLM workloads and the corresponding software ecosystem are not fully mature and streamlined, given the requirement of domain-specific tuning. This paper aims at filling this gap, focusing on optimizing LLM inference on the Sophon SG2042, the first commercially available many-core RISC-V CPU with vector processing capabilities.
On two recent state-of-the-art LLMs optimized for reasoning, DeepSeek R1 Distill Llama 8B and DeepSeek R1 Distill QWEN 14B, we achieve 4.32/2.29 token/s for token generation and 6.54/3.68 token/s for prompt processing, with a speed up of up 2.9x/3.0x compared to our baseline. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_17422 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | V-Seek: Accelerating LLM Reasoning on Open-hardware Server-class RISC-V Platforms Rodrigo, Javier J. Poveda Ahmdi, Mohamed Amine Burrello, Alessio Pagliari, Daniele Jahier Benini, Luca Machine Learning Performance The recent exponential growth of Large Language Models (LLMs) has relied on GPU-based systems. However, CPUs are emerging as a flexible and lower-cost alternative, especially when targeting inference and reasoning workloads. RISC-V is rapidly gaining traction in this area, given its open and vendor-neutral ISA. However, the RISC-V hardware for LLM workloads and the corresponding software ecosystem are not fully mature and streamlined, given the requirement of domain-specific tuning. This paper aims at filling this gap, focusing on optimizing LLM inference on the Sophon SG2042, the first commercially available many-core RISC-V CPU with vector processing capabilities. On two recent state-of-the-art LLMs optimized for reasoning, DeepSeek R1 Distill Llama 8B and DeepSeek R1 Distill QWEN 14B, we achieve 4.32/2.29 token/s for token generation and 6.54/3.68 token/s for prompt processing, with a speed up of up 2.9x/3.0x compared to our baseline. |
| title | V-Seek: Accelerating LLM Reasoning on Open-hardware Server-class RISC-V Platforms |
| topic | Machine Learning Performance |
| url | https://arxiv.org/abs/2503.17422 |