Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.07710 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908707829317632 |
|---|---|
| author | Wang, Jingya Deng, Xin Wei, Wenjie Zhang, Dehao Wang, Shuai Sun, Qian Zhang, Jieyuan Liu, Hanwen Xie, Ning Zhang, Malu |
| author_facet | Wang, Jingya Deng, Xin Wei, Wenjie Zhang, Dehao Wang, Shuai Sun, Qian Zhang, Jieyuan Liu, Hanwen Xie, Ning Zhang, Malu |
| contents | Leveraging the event-driven paradigm, Spiking Neural Networks (SNNs) offer a promising approach for energy-efficient Transformer architectures.While ANN-to-SNN conversion avoids the high training cost of directly trained Spiking Transformers, existing approaches still struggle to handle the nonlinear operations within Transformer blocks, and often require additional fine-tuning of pretrained ANNs.To address these limitations, we propose a training-free and high-performance ANN-to-SNN conversion framework tailored for Transformer architectures. Specifically, we introduce a Multi-basis Exponential Decay (MBE) neuron that combines exponential decay with a multi-basis encoding strategy to effectively approximate nonlinear operations, eliminating the need for weight modifications in pretrained ANNs.Extensive experiments across diverse tasks (CV, NLU, NLG) and mainstream Transformer architectures (ViT, RoBERTa, GPT-2) demonstrate that our method achieves near-lossless conversion accuracy with significantly lower latency. This provides a promising pathway for the efficient and scalable deployment of Spiking Transformers in real-world applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_07710 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Training-Free ANN-to-SNN Conversion for High-Performance Spiking Transformer Wang, Jingya Deng, Xin Wei, Wenjie Zhang, Dehao Wang, Shuai Sun, Qian Zhang, Jieyuan Liu, Hanwen Xie, Ning Zhang, Malu Machine Learning Artificial Intelligence Leveraging the event-driven paradigm, Spiking Neural Networks (SNNs) offer a promising approach for energy-efficient Transformer architectures.While ANN-to-SNN conversion avoids the high training cost of directly trained Spiking Transformers, existing approaches still struggle to handle the nonlinear operations within Transformer blocks, and often require additional fine-tuning of pretrained ANNs.To address these limitations, we propose a training-free and high-performance ANN-to-SNN conversion framework tailored for Transformer architectures. Specifically, we introduce a Multi-basis Exponential Decay (MBE) neuron that combines exponential decay with a multi-basis encoding strategy to effectively approximate nonlinear operations, eliminating the need for weight modifications in pretrained ANNs.Extensive experiments across diverse tasks (CV, NLU, NLG) and mainstream Transformer architectures (ViT, RoBERTa, GPT-2) demonstrate that our method achieves near-lossless conversion accuracy with significantly lower latency. This provides a promising pathway for the efficient and scalable deployment of Spiking Transformers in real-world applications. |
| title | Training-Free ANN-to-SNN Conversion for High-Performance Spiking Transformer |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2508.07710 |