FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917363340804096 |
|---|---|
| author | Liu, Dong Yu, Yanxuan Zhang, Jiayi Li, Yifan Lengerich, Ben Wu, Ying Nian |
| author_facet | Liu, Dong Yu, Yanxuan Zhang, Jiayi Li, Yifan Lengerich, Ben Wu, Ying Nian |
| contents | Diffusion Transformers (DiT) are powerful generative models but remain computationally intensive due to their iterative structure and deep transformer stacks.
To alleviate this inefficiency, we propose \textbf{FastCache}, a hidden-state-level caching and compression framework that accelerates DiT inference by exploiting redundancy within the model's internal representations.
FastCache introduces a dual strategy: (1) a spatial-aware token selection mechanism that adaptively filters redundant tokens based on hidden-state saliency, and (2) a transformer-level cache that reuses latent activations across timesteps when changes fall below a predefined threshold.
These modules work jointly to reduce unnecessary computation while preserving generation fidelity through learnable linear approximation.
Theoretical analysis shows that FastCache maintains bounded approximation error under a hypothesis-testing-based decision rule.
Empirical evaluations across multiple DiT variants demonstrate substantial reductions in latency and memory usage, achieving the best generation quality among existing cache methods, as measured by FID and t-FID.
To further improve the speedup of FastCache, we also introduce a token merging module that merges redundant tokens based on k-NN density.
Code is available at \href{https://github.com/NoakLiu/FastCache-xDiT}{https://github.com/NoakLiu/FastCache-xDiT}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_20353 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximation Liu, Dong Yu, Yanxuan Zhang, Jiayi Li, Yifan Lengerich, Ben Wu, Ying Nian Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Multimedia Performance Diffusion Transformers (DiT) are powerful generative models but remain computationally intensive due to their iterative structure and deep transformer stacks. To alleviate this inefficiency, we propose \textbf{FastCache}, a hidden-state-level caching and compression framework that accelerates DiT inference by exploiting redundancy within the model's internal representations. FastCache introduces a dual strategy: (1) a spatial-aware token selection mechanism that adaptively filters redundant tokens based on hidden-state saliency, and (2) a transformer-level cache that reuses latent activations across timesteps when changes fall below a predefined threshold. These modules work jointly to reduce unnecessary computation while preserving generation fidelity through learnable linear approximation. Theoretical analysis shows that FastCache maintains bounded approximation error under a hypothesis-testing-based decision rule. Empirical evaluations across multiple DiT variants demonstrate substantial reductions in latency and memory usage, achieving the best generation quality among existing cache methods, as measured by FID and t-FID. To further improve the speedup of FastCache, we also introduce a token merging module that merges redundant tokens based on k-NN density. Code is available at \href{https://github.com/NoakLiu/FastCache-xDiT}{https://github.com/NoakLiu/FastCache-xDiT}. |
| title | FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximation |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Multimedia Performance |
| url | https://arxiv.org/abs/2505.20353 |