Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2606.00539 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917550205435904 |
|---|---|
| author | Kong, Boao Jia, Weichen Zhang, Engao Li, Guohong Dong, Yonghan Wang, Yao Wang, Yaoyuan Peng, Yunke Yuan, Kun |
| author_facet | Kong, Boao Jia, Weichen Zhang, Engao Li, Guohong Dong, Yonghan Wang, Yao Wang, Yaoyuan Peng, Yunke Yuan, Kun |
| contents | Training stability is a key bottleneck in low-precision language model training: efficient low-cost paths can still produce short-lived numerical risks at a small set of operators. We formulate this as runtime stability control and present Gradient Norm-to-Mean Ratio (GNMR), a lightweight controller that compares each recoverable unit's current gradient norm with its historical mean. Together with $Δ$-GNMR for abrupt short-window increases, GNMR maps local risk signals to bounded recovery actions under a hard $\mathrm{maxO}$ budget and a short lock interval, without changing the numerical format, kernel, or backend recipe. Across activation-quantization stress, DeepSeek-style recipe-level training, and LLaMA-2 13B fine-tuning, GNMR preserves high-fidelity quality with sparse, budgeted recovery. These results support GNMR as a backend-agnostic controller to improve low-precision training stability while preserving low-cost execution. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_00539 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | GNMR: Runtime Stability Control for Low-Precision Large Language Model Training Kong, Boao Jia, Weichen Zhang, Engao Li, Guohong Dong, Yonghan Wang, Yao Wang, Yaoyuan Peng, Yunke Yuan, Kun Machine Learning Optimization and Control Training stability is a key bottleneck in low-precision language model training: efficient low-cost paths can still produce short-lived numerical risks at a small set of operators. We formulate this as runtime stability control and present Gradient Norm-to-Mean Ratio (GNMR), a lightweight controller that compares each recoverable unit's current gradient norm with its historical mean. Together with $Δ$-GNMR for abrupt short-window increases, GNMR maps local risk signals to bounded recovery actions under a hard $\mathrm{maxO}$ budget and a short lock interval, without changing the numerical format, kernel, or backend recipe. Across activation-quantization stress, DeepSeek-style recipe-level training, and LLaMA-2 13B fine-tuning, GNMR preserves high-fidelity quality with sparse, budgeted recovery. These results support GNMR as a backend-agnostic controller to improve low-precision training stability while preserving low-cost execution. |
| title | GNMR: Runtime Stability Control for Low-Precision Large Language Model Training |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2606.00539 |