Exploring the Benefit of Activation Sparsity in Pre-training
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910633220374528 |
|---|---|
| author | Zhang, Zhengyan Xiao, Chaojun Qin, Qiujieli Lin, Yankai Zeng, Zhiyuan Han, Xu Liu, Zhiyuan Xie, Ruobing Sun, Maosong Zhou, Jie |
| author_facet | Zhang, Zhengyan Xiao, Chaojun Qin, Qiujieli Lin, Yankai Zeng, Zhiyuan Han, Xu Liu, Zhiyuan Xie, Ruobing Sun, Maosong Zhou, Jie |
| contents | Pre-trained Transformers inherently possess the characteristic of sparse activation, where only a small fraction of the neurons are activated for each token. While sparse activation has been explored through post-training methods, its potential in pre-training remains untapped. In this work, we first study how activation properties change during pre-training. Our examination reveals that Transformers exhibit sparse activation throughout the majority of the pre-training process while the activation correlation keeps evolving as training progresses. Leveraging this observation, we propose Switchable Sparse-Dense Learning (SSD). SSD adaptively switches between the Mixtures-of-Experts (MoE) based sparse training and the conventional dense training during the pre-training process, leveraging the efficiency of sparse training and avoiding the static activation correlation of sparse training. Compared to dense training, SSD achieves comparable performance with identical model size and reduces pre-training costs. Moreover, the models trained with SSD can be directly used as MoE models for sparse inference and achieve the same performance as dense models with up to $2\times$ faster inference speed. Codes are available at https://github.com/thunlp/moefication. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_03440 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Exploring the Benefit of Activation Sparsity in Pre-training Zhang, Zhengyan Xiao, Chaojun Qin, Qiujieli Lin, Yankai Zeng, Zhiyuan Han, Xu Liu, Zhiyuan Xie, Ruobing Sun, Maosong Zhou, Jie Computation and Language Artificial Intelligence Pre-trained Transformers inherently possess the characteristic of sparse activation, where only a small fraction of the neurons are activated for each token. While sparse activation has been explored through post-training methods, its potential in pre-training remains untapped. In this work, we first study how activation properties change during pre-training. Our examination reveals that Transformers exhibit sparse activation throughout the majority of the pre-training process while the activation correlation keeps evolving as training progresses. Leveraging this observation, we propose Switchable Sparse-Dense Learning (SSD). SSD adaptively switches between the Mixtures-of-Experts (MoE) based sparse training and the conventional dense training during the pre-training process, leveraging the efficiency of sparse training and avoiding the static activation correlation of sparse training. Compared to dense training, SSD achieves comparable performance with identical model size and reduces pre-training costs. Moreover, the models trained with SSD can be directly used as MoE models for sparse inference and achieve the same performance as dense models with up to $2\times$ faster inference speed. Codes are available at https://github.com/thunlp/moefication. |
| title | Exploring the Benefit of Activation Sparsity in Pre-training |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2410.03440 |