Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2502.17055 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918525206003712 |
|---|---|
| author | Huang, Tianjin Wang, Zhangyang Hu, Haotian Zhang, Zhenyu Jin, Gaojie Li, Xiang Shen, Li Shang, Jiaxing Chen, Tianlong Li, Ke Liu, Lu Wen, Qingsong Liu, Shiwei |
| author_facet | Huang, Tianjin Wang, Zhangyang Hu, Haotian Zhang, Zhenyu Jin, Gaojie Li, Xiang Shen, Li Shang, Jiaxing Chen, Tianlong Li, Ke Liu, Lu Wen, Qingsong Liu, Shiwei |
| contents | Training instability in modern deep learning systems is frequently triggered by rare but extreme gradient-norm spikes, which can induce oversized parameter updates, corrupt optimizer state, and lead to slow recovery or divergence. Widely used safeguards such as gradient clipping mitigate these failures but require threshold tuning and indiscriminately truncate large updates. We propose GradientStabilizer, a lightweight, drop-in gradient transform that preserves the instantaneous gradient direction while replacing the update magnitude with a statistically stabilized estimate derived from running gradient-norm statistics. We prove that the resulting stabilized magnitude is uniformly bounded on spike steps, independent of the spike size, and show how this boundedness controls optimizer state evolution in adaptive methods. Across LLM pre-training (FP16), quantization-aware pre-training (FP4), ImageNet classification, reinforcement learning, and time-series forecasting, GradientStabilizer consistently improves training stability, widens stable learning-rate regions, and reduces divergence relative to clipping-based baselines, even substantially reducing Adam's sensitivity to weight-decay strength. Code will be released soon. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_17055 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GradientStabilizer:Fix the Norm, Not the Gradient Huang, Tianjin Wang, Zhangyang Hu, Haotian Zhang, Zhenyu Jin, Gaojie Li, Xiang Shen, Li Shang, Jiaxing Chen, Tianlong Li, Ke Liu, Lu Wen, Qingsong Liu, Shiwei Machine Learning Artificial Intelligence Training instability in modern deep learning systems is frequently triggered by rare but extreme gradient-norm spikes, which can induce oversized parameter updates, corrupt optimizer state, and lead to slow recovery or divergence. Widely used safeguards such as gradient clipping mitigate these failures but require threshold tuning and indiscriminately truncate large updates. We propose GradientStabilizer, a lightweight, drop-in gradient transform that preserves the instantaneous gradient direction while replacing the update magnitude with a statistically stabilized estimate derived from running gradient-norm statistics. We prove that the resulting stabilized magnitude is uniformly bounded on spike steps, independent of the spike size, and show how this boundedness controls optimizer state evolution in adaptive methods. Across LLM pre-training (FP16), quantization-aware pre-training (FP4), ImageNet classification, reinforcement learning, and time-series forecasting, GradientStabilizer consistently improves training stability, widens stable learning-rate regions, and reduces divergence relative to clipping-based baselines, even substantially reducing Adam's sensitivity to weight-decay strength. Code will be released soon. |
| title | GradientStabilizer:Fix the Norm, Not the Gradient |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2502.17055 |