Mapping Gemma3 onto an Edge Dataflow Architecture
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910030972846080 |
|---|---|
| author | Du, Shouyu Yu, Miaoxiang Xu, Zhenyu Ni, Zhiheng Cai, Jillian Yang, Qing Wei, Tao |
| author_facet | Du, Shouyu Yu, Miaoxiang Xu, Zhenyu Ni, Zhiheng Cai, Jillian Yang, Qing Wei, Tao |
| contents | We present the first end-to-end deployment of the Gemma3 family of large language and vision models on a tiled edge dataflow architecture (AMD Ryzen AI NPU). Our work introduces a set of hardware-aware techniques. For prefill, we introduce an efficient dequantization engine, optimize tiled matrix multiplication kernels, and propose FlowQKV, a chunked, pipelined attention mechanism. For decoding, we introduce FusedDQP, which fuses dequantization and projection into a single kernel, and FlowKV, which re-structures attention to sustain high memory bandwidth utilization. Together with a compact Q4NX 4-bit quantization format, these methods yield up to $5.2\times$ faster prefill and $4.8\times$ faster decoding versus the iGPU, and $33.5\times$ and $2.2\times$ over the CPU, respectively. Power efficiency improves by as much as $67.2\times$ and $222.9\times$ compared to the iGPU and CPU. The proposed approach demonstrates that modern NPUs can deliver practical, low-power LLM and VLM inference at the edge, and provides a generalizable blueprint for mapping transformer-based models onto tiled dataflow accelerators. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_06063 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Mapping Gemma3 onto an Edge Dataflow Architecture Du, Shouyu Yu, Miaoxiang Xu, Zhenyu Ni, Zhiheng Cai, Jillian Yang, Qing Wei, Tao Distributed, Parallel, and Cluster Computing We present the first end-to-end deployment of the Gemma3 family of large language and vision models on a tiled edge dataflow architecture (AMD Ryzen AI NPU). Our work introduces a set of hardware-aware techniques. For prefill, we introduce an efficient dequantization engine, optimize tiled matrix multiplication kernels, and propose FlowQKV, a chunked, pipelined attention mechanism. For decoding, we introduce FusedDQP, which fuses dequantization and projection into a single kernel, and FlowKV, which re-structures attention to sustain high memory bandwidth utilization. Together with a compact Q4NX 4-bit quantization format, these methods yield up to $5.2\times$ faster prefill and $4.8\times$ faster decoding versus the iGPU, and $33.5\times$ and $2.2\times$ over the CPU, respectively. Power efficiency improves by as much as $67.2\times$ and $222.9\times$ compared to the iGPU and CPU. The proposed approach demonstrates that modern NPUs can deliver practical, low-power LLM and VLM inference at the edge, and provides a generalizable blueprint for mapping transformer-based models onto tiled dataflow accelerators. |
| title | Mapping Gemma3 onto an Edge Dataflow Architecture |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2602.06063 |