Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.00486 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918348788334592 |
|---|---|
| author | Wu, Jun Huang, Patrick Wen, Jiangtao Han, Yuxing |
| author_facet | Wu, Jun Huang, Patrick Wen, Jiangtao Han, Yuxing |
| contents | Despite rapid progress in large language models (LLMs), the statistical structure of their weights, activations, and gradients-and its implications for initialization, training dynamics, and efficiency-remains largely unexplored. We empirically show that these quantities in LLMs are well modeled by generalized Gaussian (GG) distributions, and introduce a unified, end-to-end optimization framework grounded in this observation. Our contributions are threefold: (1) a GG-based initialization that aligns with trained model statistics, accelerating convergence and improving accuracy; (2) ACT, a progressive activation-constrained training method that reduces redundancy and propagation overhead; and (3) GCT, a gradient-constrained training algorithm that substantially lowers communication cost in distributed training. Experiments across diverse architectures demonstrate consistently smaller, faster models with minimal communication overhead that match or surpass standard baselines. By anchoring LLM optimization in principled statistical modeling, this work advances efficient, scalable, and hardware-aware AI systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_00486 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | It Takes a Good Model to Train a Good Model: Generalized Gaussian Priors for Optimized LLMs Wu, Jun Huang, Patrick Wen, Jiangtao Han, Yuxing Machine Learning Artificial Intelligence Despite rapid progress in large language models (LLMs), the statistical structure of their weights, activations, and gradients-and its implications for initialization, training dynamics, and efficiency-remains largely unexplored. We empirically show that these quantities in LLMs are well modeled by generalized Gaussian (GG) distributions, and introduce a unified, end-to-end optimization framework grounded in this observation. Our contributions are threefold: (1) a GG-based initialization that aligns with trained model statistics, accelerating convergence and improving accuracy; (2) ACT, a progressive activation-constrained training method that reduces redundancy and propagation overhead; and (3) GCT, a gradient-constrained training algorithm that substantially lowers communication cost in distributed training. Experiments across diverse architectures demonstrate consistently smaller, faster models with minimal communication overhead that match or surpass standard baselines. By anchoring LLM optimization in principled statistical modeling, this work advances efficient, scalable, and hardware-aware AI systems. |
| title | It Takes a Good Model to Train a Good Model: Generalized Gaussian Priors for Optimized LLMs |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2506.00486 |