Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.25704 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911715834200064 |
|---|---|
| author | Jiang, Peijie Feng, Yuqi Peng, Cunyin Zhao, Qian Liu, Jia Chen, KunLong Zhang, Zhiqiang Zhou, Jun |
| author_facet | Jiang, Peijie Feng, Yuqi Peng, Cunyin Zhao, Qian Liu, Jia Chen, KunLong Zhang, Zhiqiang Zhou, Jun |
| contents | In contemporary large language models (LLMs), the swish-gated linear unit (SwiGLU) activation function is widely adopted to regulate the information flow and introduce non-linearity. For large positive inputs, SwiGLU approximates the quadratic function $x^2$, providing strong nonlinearity and expressive capacity. However, this property also causes numerical instability as the input or model scale increases, particularly in low-precision LLM training. The main reason is its approximate quadratic amplification, which enlarges the output range and exacerbates outliers. To address this issue, we propose a stable activation function, Power Linear Unit (PowLU), for large-scale LLM pre-training. Specifically, PowLU employs a rational power function to achieve adaptive nonlinearity, thereby improving representation ability and enabling stable training in spike regions. Moreover, we provide theoretical justification for several key properties of PowLU. Scaling law experiments confirm that the performance is consistent across model sizes, and further experimental results with the Ling architecture (7.9B and 124B total parameters) demonstrate that PowLU achieves competitive results against SwiGLU and SwiGLU-Clip in large-scale training of LLMs. In addition, the experimental results also show that PowLU effectively improves the scalability of the large-scale training of LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_25704 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | PowLU: An Activation Function for Stable Pre-Training of LLMs Jiang, Peijie Feng, Yuqi Peng, Cunyin Zhao, Qian Liu, Jia Chen, KunLong Zhang, Zhiqiang Zhou, Jun Computation and Language Machine Learning I.2.7 In contemporary large language models (LLMs), the swish-gated linear unit (SwiGLU) activation function is widely adopted to regulate the information flow and introduce non-linearity. For large positive inputs, SwiGLU approximates the quadratic function $x^2$, providing strong nonlinearity and expressive capacity. However, this property also causes numerical instability as the input or model scale increases, particularly in low-precision LLM training. The main reason is its approximate quadratic amplification, which enlarges the output range and exacerbates outliers. To address this issue, we propose a stable activation function, Power Linear Unit (PowLU), for large-scale LLM pre-training. Specifically, PowLU employs a rational power function to achieve adaptive nonlinearity, thereby improving representation ability and enabling stable training in spike regions. Moreover, we provide theoretical justification for several key properties of PowLU. Scaling law experiments confirm that the performance is consistent across model sizes, and further experimental results with the Ling architecture (7.9B and 124B total parameters) demonstrate that PowLU achieves competitive results against SwiGLU and SwiGLU-Clip in large-scale training of LLMs. In addition, the experimental results also show that PowLU effectively improves the scalability of the large-scale training of LLMs. |
| title | PowLU: An Activation Function for Stable Pre-Training of LLMs |
| topic | Computation and Language Machine Learning I.2.7 |
| url | https://arxiv.org/abs/2605.25704 |