Stochastic activations
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908730997604352 |
|---|---|
| author | Lomeli, Maria Douze, Matthijs Szilvasy, Gergely Cabannes, Loic Copet, Jade Sukhbaatar, Sainbayar Weston, Jason Synnaeve, Gabriel Mazaré, Pierre-Emmanuel Jégou, Hervé |
| author_facet | Lomeli, Maria Douze, Matthijs Szilvasy, Gergely Cabannes, Loic Copet, Jade Sukhbaatar, Sainbayar Weston, Jason Synnaeve, Gabriel Mazaré, Pierre-Emmanuel Jégou, Hervé |
| contents | We introduce stochastic activations. This novel strategy randomly selects between several non-linear functions in the feed-forward layer of a large language model. In particular, we choose between SILU or RELU depending on a Bernoulli draw. This strategy circumvents the optimization problem associated with RELU, namely, the constant shape for negative inputs that prevents the gradient flow. We leverage this strategy in two ways:
(1) We use stochastic activations during pre-training and fine-tune the model with RELU, which is used at inference time to provide sparse latent vectors. This reduces the inference FLOPs and translates into a significant speedup on CPU and GPU. This leads to better results than training from scratch with the RELU activation function.
(2) We evaluate stochastic activations for sequence generation. This strategy performs reasonably well: it has higher diversity and has only slightly inferior performance to the best deterministic non-linearity, SILU, combined with temperature sampling. This provides an alternative way to increase the diversity of generated text. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_22358 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Stochastic activations Lomeli, Maria Douze, Matthijs Szilvasy, Gergely Cabannes, Loic Copet, Jade Sukhbaatar, Sainbayar Weston, Jason Synnaeve, Gabriel Mazaré, Pierre-Emmanuel Jégou, Hervé Machine Learning Artificial Intelligence We introduce stochastic activations. This novel strategy randomly selects between several non-linear functions in the feed-forward layer of a large language model. In particular, we choose between SILU or RELU depending on a Bernoulli draw. This strategy circumvents the optimization problem associated with RELU, namely, the constant shape for negative inputs that prevents the gradient flow. We leverage this strategy in two ways: (1) We use stochastic activations during pre-training and fine-tune the model with RELU, which is used at inference time to provide sparse latent vectors. This reduces the inference FLOPs and translates into a significant speedup on CPU and GPU. This leads to better results than training from scratch with the RELU activation function. (2) We evaluate stochastic activations for sequence generation. This strategy performs reasonably well: it has higher diversity and has only slightly inferior performance to the best deterministic non-linearity, SILU, combined with temperature sampling. This provides an alternative way to increase the diversity of generated text. |
| title | Stochastic activations |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2509.22358 |