TRINE: A Token-Aware, Runtime-Adaptive FPGA Inference Engine for Multimodal AI
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918530870411264 |
|---|---|
| author | Oh, Hyunwoo Chen, Hanning Yun, Sanggeon Ni, Yang Jang, Suyeon Khaleghi, Behnam Wen, Fei Imani, Mohsen |
| author_facet | Oh, Hyunwoo Chen, Hanning Yun, Sanggeon Ni, Yang Jang, Suyeon Khaleghi, Behnam Wen, Fei Imani, Mohsen |
| contents | Multimodal stacks that mix ViTs, CNNs, GNNs, and transformer NLP strain embedded platforms because their compute/memory patterns diverge and hard real-time targets leave little slack. TRINE is a single-bitstream FPGA accelerator and compiler that executes end-to-end multimodal inference without reconfiguration. Layers are unified as DDMM/SDDMM/SpMM and mapped to a mode-switchable engine that toggles at runtime among weight/output-stationary systolic, 1xCS SIMD, and a routable adder tree (RADT) on a shared PE array. A width-matched, two-stage top-k unit enables in-stream token pruning, while dependency-aware layer offloading (DALO) overlaps independent kernels across reconfigurable processing units to sustain utilization. Evaluated on Alveo U50 and ZCU104, TRINE reduces latency by up to 22.57x vs. RTX 4090 and 6.86x vs. Jetson Orin Nano at 20-21 W; token pruning alone yields up to 7.8x on ViT-heavy pipelines, and DALO contributes up to 79% throughput improvement. With int8 quantization, accuracy drops remain <2.5% across representative tasks, delivering state-of-the-art latency and energy efficiency for unified vision, language, and graph workloads-in one bitstream. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_22867 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | TRINE: A Token-Aware, Runtime-Adaptive FPGA Inference Engine for Multimodal AI Oh, Hyunwoo Chen, Hanning Yun, Sanggeon Ni, Yang Jang, Suyeon Khaleghi, Behnam Wen, Fei Imani, Mohsen Hardware Architecture Artificial Intelligence Machine Learning Multimodal stacks that mix ViTs, CNNs, GNNs, and transformer NLP strain embedded platforms because their compute/memory patterns diverge and hard real-time targets leave little slack. TRINE is a single-bitstream FPGA accelerator and compiler that executes end-to-end multimodal inference without reconfiguration. Layers are unified as DDMM/SDDMM/SpMM and mapped to a mode-switchable engine that toggles at runtime among weight/output-stationary systolic, 1xCS SIMD, and a routable adder tree (RADT) on a shared PE array. A width-matched, two-stage top-k unit enables in-stream token pruning, while dependency-aware layer offloading (DALO) overlaps independent kernels across reconfigurable processing units to sustain utilization. Evaluated on Alveo U50 and ZCU104, TRINE reduces latency by up to 22.57x vs. RTX 4090 and 6.86x vs. Jetson Orin Nano at 20-21 W; token pruning alone yields up to 7.8x on ViT-heavy pipelines, and DALO contributes up to 79% throughput improvement. With int8 quantization, accuracy drops remain <2.5% across representative tasks, delivering state-of-the-art latency and energy efficiency for unified vision, language, and graph workloads-in one bitstream. |
| title | TRINE: A Token-Aware, Runtime-Adaptive FPGA Inference Engine for Multimodal AI |
| topic | Hardware Architecture Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2603.22867 |