To 2:4 Sparsity and Beyond: Neuron-level Activation Function to Accelerate LLM Pre-Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910013421780992 |
|---|---|
| author | Madhyastha, Meghana Haziza, Daniel Cai, Jesse Ardalani, Newsha Bu, Zhiqi Wu, Carole-Jean |
| author_facet | Madhyastha, Meghana Haziza, Daniel Cai, Jesse Ardalani, Newsha Bu, Zhiqi Wu, Carole-Jean |
| contents | Trainings of Large Language Models are generally bottlenecked by matrix multiplications. In the Transformer architecture, a large portion of these operations happens in the Feed Forward Network (FFN), and this portion increases for larger models, up to 50% of the total pretraining floating point operations. We show that we can leverage hardware-accelerated sparsity to accelerate all matrix multiplications in the FFN, with 2:4 sparsity for weights and v:n:m (Venom) sparsity for activations. Our recipe relies on sparse training steps to accelerate a large part of the pretraining, associated with regular dense training steps towards the end. Overall, models trained with this approach exhibit the same performance on our quality benchmarks, and can speed up training end-to-end by 1.4 to 1.7x. This approach is applicable to all NVIDIA GPUs starting with the A100 generation, and is orthogonal to common optimization techniques, such as, quantization, and can also be applied to mixture-of-experts model architectures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_06183 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | To 2:4 Sparsity and Beyond: Neuron-level Activation Function to Accelerate LLM Pre-Training Madhyastha, Meghana Haziza, Daniel Cai, Jesse Ardalani, Newsha Bu, Zhiqi Wu, Carole-Jean Machine Learning Trainings of Large Language Models are generally bottlenecked by matrix multiplications. In the Transformer architecture, a large portion of these operations happens in the Feed Forward Network (FFN), and this portion increases for larger models, up to 50% of the total pretraining floating point operations. We show that we can leverage hardware-accelerated sparsity to accelerate all matrix multiplications in the FFN, with 2:4 sparsity for weights and v:n:m (Venom) sparsity for activations. Our recipe relies on sparse training steps to accelerate a large part of the pretraining, associated with regular dense training steps towards the end. Overall, models trained with this approach exhibit the same performance on our quality benchmarks, and can speed up training end-to-end by 1.4 to 1.7x. This approach is applicable to all NVIDIA GPUs starting with the A100 generation, and is orthogonal to common optimization techniques, such as, quantization, and can also be applied to mixture-of-experts model architectures. |
| title | To 2:4 Sparsity and Beyond: Neuron-level Activation Function to Accelerate LLM Pre-Training |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2602.06183 |