Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.21503 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913052933226496 |
|---|---|
| author | Cai, Junhong Wang, Guiqin Zhao, Kejie Tang, Jianxiong Wang, Xiang Leng, Luziwei Cheng, Ran Ma, Yuxin Guo, Qinghai |
| author_facet | Cai, Junhong Wang, Guiqin Zhao, Kejie Tang, Jianxiong Wang, Xiang Leng, Luziwei Cheng, Ran Ma, Yuxin Guo, Qinghai |
| contents | Large Language Models (LLMs) excel across diverse domains but suffer from high energy costs due to quadratic attention and dense Feed-Forward Network (FFN) operations. To address these issues, we propose Module-aware Architecture Refinement (MAR), a two-stage framework that integrates State Space Models (SSMs) for linear-time sequence modeling and applies activation sparsification to reduce FFN costs. In addition, to mitigate low information density and temporal mismatch in integrating Spiking Neural Networks (SNNs) with SSMs, we design the Adaptive Ternary Multi-step Neuron (ATMN) and the Spike-aware Bidirectional Distillation Strategy (SBDS). Extensive experiments demonstrate that MAR effectively restores the performance of its dense counterpart under constrained resources while substantially reducing inference energy consumption. Furthermore, it outperforms efficient models of comparable or even larger scale, underscoring its potential for building efficient and practical LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_21503 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MAR: Efficient Large Language Models via Module-aware Architecture Refinement Cai, Junhong Wang, Guiqin Zhao, Kejie Tang, Jianxiong Wang, Xiang Leng, Luziwei Cheng, Ran Ma, Yuxin Guo, Qinghai Artificial Intelligence Computation and Language Machine Learning Neural and Evolutionary Computing Large Language Models (LLMs) excel across diverse domains but suffer from high energy costs due to quadratic attention and dense Feed-Forward Network (FFN) operations. To address these issues, we propose Module-aware Architecture Refinement (MAR), a two-stage framework that integrates State Space Models (SSMs) for linear-time sequence modeling and applies activation sparsification to reduce FFN costs. In addition, to mitigate low information density and temporal mismatch in integrating Spiking Neural Networks (SNNs) with SSMs, we design the Adaptive Ternary Multi-step Neuron (ATMN) and the Spike-aware Bidirectional Distillation Strategy (SBDS). Extensive experiments demonstrate that MAR effectively restores the performance of its dense counterpart under constrained resources while substantially reducing inference energy consumption. Furthermore, it outperforms efficient models of comparable or even larger scale, underscoring its potential for building efficient and practical LLMs. |
| title | MAR: Efficient Large Language Models via Module-aware Architecture Refinement |
| topic | Artificial Intelligence Computation and Language Machine Learning Neural and Evolutionary Computing |
| url | https://arxiv.org/abs/2601.21503 |