Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.14709 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918027657740288 |
|---|---|
| author | Shen, Xuan Ma, Weize Zhou, Yufa Tang, Enhao Xie, Yanyue Li, Zhengang Gong, Yifan Wang, Quanyi Ding, Henghui Wang, Yiwei Wang, Yanzhi Zhao, Pu Lin, Jun Gu, Jiuxiang |
| author_facet | Shen, Xuan Ma, Weize Zhou, Yufa Tang, Enhao Xie, Yanyue Li, Zhengang Gong, Yifan Wang, Quanyi Ding, Henghui Wang, Yiwei Wang, Yanzhi Zhao, Pu Lin, Jun Gu, Jiuxiang |
| contents | Auto-regressive (AR) models, initially successful in language generation, have recently shown promise in visual generation tasks due to their superior sampling efficiency. Unlike image generation, video generation requires a substantially larger number of tokens to produce coherent temporal frames, resulting in significant overhead during the decoding phase. Our key observations are: (i) MLP modules in the decode phase dominate the inference latency, and (ii) there exists high temporal redundancy in MLP outputs of adjacent frames. In this paper, we propose the \textbf{FastCar} framework to accelerate the decode phase for the AR video generation by exploring the temporal redundancy. The Temporal Attention Score (TAS) is proposed to determine whether to apply the replay strategy (\textit{i.e.}, reusing cached MLP outputs from the previous frame to reduce redundant computations) with detailed theoretical analysis and justification. Also, we develop a hardware accelerator on FPGA with Dynamic Resource Scheduling (DRS) based on TAS to enable better resource utilization and faster inference. Experimental results demonstrate the effectiveness of our method, which outperforms traditional sparse attention approaches with more than 2.1x decoding speedup and higher energy efficiency on the edge. Furthermore, by combining FastCar and sparse attention, FastCar can boost the performance of sparse attention with alleviated drifting, demonstrating our unique advantages for high-resolution and long-duration video generation. Code: https://github.com/shawnricecake/fast-car |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_14709 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FastCar: Cache Attentive Replay for Fast Auto-Regressive Video Generation on the Edge Shen, Xuan Ma, Weize Zhou, Yufa Tang, Enhao Xie, Yanyue Li, Zhengang Gong, Yifan Wang, Quanyi Ding, Henghui Wang, Yiwei Wang, Yanzhi Zhao, Pu Lin, Jun Gu, Jiuxiang Computer Vision and Pattern Recognition Artificial Intelligence Auto-regressive (AR) models, initially successful in language generation, have recently shown promise in visual generation tasks due to their superior sampling efficiency. Unlike image generation, video generation requires a substantially larger number of tokens to produce coherent temporal frames, resulting in significant overhead during the decoding phase. Our key observations are: (i) MLP modules in the decode phase dominate the inference latency, and (ii) there exists high temporal redundancy in MLP outputs of adjacent frames. In this paper, we propose the \textbf{FastCar} framework to accelerate the decode phase for the AR video generation by exploring the temporal redundancy. The Temporal Attention Score (TAS) is proposed to determine whether to apply the replay strategy (\textit{i.e.}, reusing cached MLP outputs from the previous frame to reduce redundant computations) with detailed theoretical analysis and justification. Also, we develop a hardware accelerator on FPGA with Dynamic Resource Scheduling (DRS) based on TAS to enable better resource utilization and faster inference. Experimental results demonstrate the effectiveness of our method, which outperforms traditional sparse attention approaches with more than 2.1x decoding speedup and higher energy efficiency on the edge. Furthermore, by combining FastCar and sparse attention, FastCar can boost the performance of sparse attention with alleviated drifting, demonstrating our unique advantages for high-resolution and long-duration video generation. Code: https://github.com/shawnricecake/fast-car |
| title | FastCar: Cache Attentive Replay for Fast Auto-Regressive Video Generation on the Edge |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2505.14709 |