LlamaF: An Efficient Llama2 Architecture Accelerator on Embedded FPGAs
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910609428185088 |
|---|---|
| author | Xu, Han Li, Yutong Ji, Shihao |
| author_facet | Xu, Han Li, Yutong Ji, Shihao |
| contents | Large language models (LLMs) have demonstrated remarkable abilities in natural language processing. However, their deployment on resource-constrained embedded devices remains difficult due to memory and computational demands. In this paper, we present an FPGA-based accelerator designed to improve LLM inference performance on embedded FPGAs. We employ post-training quantization to reduce model size and optimize for off-chip memory bandwidth. Our design features asynchronous computation and a fully pipelined accelerator for matrix-vector multiplication. Experiments of the TinyLlama 1.1B model on a Xilinx ZCU102 platform show a 14.3-15.8x speedup and a 6.1x power efficiency improvement over running exclusively on ZCU102 processing system (PS). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_11424 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LlamaF: An Efficient Llama2 Architecture Accelerator on Embedded FPGAs Xu, Han Li, Yutong Ji, Shihao Hardware Architecture Large language models (LLMs) have demonstrated remarkable abilities in natural language processing. However, their deployment on resource-constrained embedded devices remains difficult due to memory and computational demands. In this paper, we present an FPGA-based accelerator designed to improve LLM inference performance on embedded FPGAs. We employ post-training quantization to reduce model size and optimize for off-chip memory bandwidth. Our design features asynchronous computation and a fully pipelined accelerator for matrix-vector multiplication. Experiments of the TinyLlama 1.1B model on a Xilinx ZCU102 platform show a 14.3-15.8x speedup and a 6.1x power efficiency improvement over running exclusively on ZCU102 processing system (PS). |
| title | LlamaF: An Efficient Llama2 Architecture Accelerator on Embedded FPGAs |
| topic | Hardware Architecture |
| url | https://arxiv.org/abs/2409.11424 |