ROMA: a Read-Only-Memory-based Accelerator for QLoRA-based On-Device LLM
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917303940022272 |
|---|---|
| author | Wang, Wenqiang Zhang, Yijia Zhang, Zikai Huo, Guanting Liang, Hao Cao, Shijie Xu, Ningyi |
| author_facet | Wang, Wenqiang Zhang, Yijia Zhang, Zikai Huo, Guanting Liang, Hao Cao, Shijie Xu, Ningyi |
| contents | As large language models (LLMs) demonstrate powerful capabilities, deploying them on edge devices has become increasingly crucial, offering advantages in privacy and real-time interaction. QLoRA has emerged as the standard approach for on-device LLMs, leveraging quantized models to reduce memory and computational costs while utilizing LoRA for task-specific adaptability. In this work, we propose ROMA, a QLoRA accelerator with a hybrid storage architecture that uses ROM for quantized base models and SRAM for LoRA weights and KV cache. Our insight is that the quantized base model is stable and converged, making it well-suited for ROM storage. Meanwhile, LoRA modules offer the flexibility to adapt to new data without requiring updates to the base model. To further reduce the area cost of ROM, we introduce a novel B-ROM design and integrate it with the compute unit to form a fused cell for efficient use of chip resources. ROMA can effectively store both a 4-bit 3B and a 2-bit 8B LLaMA model entirely on-chip, achieving a notable generation speed exceeding 20,000 tokens/s without requiring external memory. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_12988 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ROMA: a Read-Only-Memory-based Accelerator for QLoRA-based On-Device LLM Wang, Wenqiang Zhang, Yijia Zhang, Zikai Huo, Guanting Liang, Hao Cao, Shijie Xu, Ningyi Hardware Architecture Artificial Intelligence As large language models (LLMs) demonstrate powerful capabilities, deploying them on edge devices has become increasingly crucial, offering advantages in privacy and real-time interaction. QLoRA has emerged as the standard approach for on-device LLMs, leveraging quantized models to reduce memory and computational costs while utilizing LoRA for task-specific adaptability. In this work, we propose ROMA, a QLoRA accelerator with a hybrid storage architecture that uses ROM for quantized base models and SRAM for LoRA weights and KV cache. Our insight is that the quantized base model is stable and converged, making it well-suited for ROM storage. Meanwhile, LoRA modules offer the flexibility to adapt to new data without requiring updates to the base model. To further reduce the area cost of ROM, we introduce a novel B-ROM design and integrate it with the compute unit to form a fused cell for efficient use of chip resources. ROMA can effectively store both a 4-bit 3B and a 2-bit 8B LLaMA model entirely on-chip, achieving a notable generation speed exceeding 20,000 tokens/s without requiring external memory. |
| title | ROMA: a Read-Only-Memory-based Accelerator for QLoRA-based On-Device LLM |
| topic | Hardware Architecture Artificial Intelligence |
| url | https://arxiv.org/abs/2503.12988 |