Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.16676 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914498526314496 |
|---|---|
| author | Li, Jiacheng Tan, Jianchao Yang, Zhidong Sun, Pingwei Huo, Feiye Qin, Jiayu Zhang, Xiangyu He, Maoxin Sun, Yerui Xie, Yuchen Tan, Guangming Jia, Weile Cai, Xunliang Zhao, Tong |
| author_facet | Li, Jiacheng Tan, Jianchao Yang, Zhidong Sun, Pingwei Huo, Feiye Qin, Jiayu Zhang, Xiangyu He, Maoxin Sun, Yerui Xie, Yuchen Tan, Guangming Jia, Weile Cai, Xunliang Zhao, Tong |
| contents | Transformer architecture gradually dominates the LLM field. Recent advances in training optimization for Transformer-based large language models (LLMs) primarily focus on architectural modifications or optimizer adjustments. However, these approaches lack systematic optimization of weight patterns during training. Weight pattern refers to the distribution and relative magnitudes of weight parameters in a neural network. To address this issue, we propose a Weight Scaling method called WISCA to enhance training efficiency and model quality by strategically improving neural network weight patterns without changing network structures. By rescaling weights while preserving model outputs, WISCA indirectly optimizes the model's training trajectory. Experiments demonstrate that WISCA significantly improves convergence quality (measured by generalization capability and loss reduction), particularly in LLMs with Grouped Query Attention (GQA) architectures and LoRA fine-tuning tasks. Empirical results show 5.6% average improvement on zero-shot validation tasks and 2.12% average reduction in training perplexity across multiple architectures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_16676 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | WISCA: A Lightweight Model Transition Method to Improve LLM Training via Weight Scaling Li, Jiacheng Tan, Jianchao Yang, Zhidong Sun, Pingwei Huo, Feiye Qin, Jiayu Zhang, Xiangyu He, Maoxin Sun, Yerui Xie, Yuchen Tan, Guangming Jia, Weile Cai, Xunliang Zhao, Tong Machine Learning Computation and Language Transformer architecture gradually dominates the LLM field. Recent advances in training optimization for Transformer-based large language models (LLMs) primarily focus on architectural modifications or optimizer adjustments. However, these approaches lack systematic optimization of weight patterns during training. Weight pattern refers to the distribution and relative magnitudes of weight parameters in a neural network. To address this issue, we propose a Weight Scaling method called WISCA to enhance training efficiency and model quality by strategically improving neural network weight patterns without changing network structures. By rescaling weights while preserving model outputs, WISCA indirectly optimizes the model's training trajectory. Experiments demonstrate that WISCA significantly improves convergence quality (measured by generalization capability and loss reduction), particularly in LLMs with Grouped Query Attention (GQA) architectures and LoRA fine-tuning tasks. Empirical results show 5.6% average improvement on zero-shot validation tasks and 2.12% average reduction in training perplexity across multiple architectures. |
| title | WISCA: A Lightweight Model Transition Method to Improve LLM Training via Weight Scaling |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2508.16676 |