Structured First-Layer Initialization Pre-Training Techniques to Accelerate Training Process Based on $\varepsilon$-Rank
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918094065106944 |
|---|---|
| author | Tang, Tao Yang, Jiang Zhao, Yuxiang Zhu, Quanhui |
| author_facet | Tang, Tao Yang, Jiang Zhao, Yuxiang Zhu, Quanhui |
| contents | Training deep neural networks for scientific computing remains computationally expensive due to the slow formation of diverse feature representations in early training stages. Recent studies identify a staircase phenomenon in training dynamics, where loss decreases are closely correlated with increases in $\varepsilon$-rank, reflecting the effective number of linearly independent neuron functions. Motivated by this observation, this work proposes a structured first-layer initialization (SFLI) pre-training method to enhance the diversity of neural features at initialization by constructing $\varepsilon$-linearly independent neurons in the input layer. We present systematic initialization schemes compatible with various activation functions and integrate the strategy into multiple neural architectures, including modified multi-layer perceptrons and physics-informed residual adaptive networks. Extensive numerical experiments on function approximation and PDE benchmarks, demonstrate that SFLI significantly improves the initial $\varepsilon$-rank, accelerates convergence, mitigates spectral bias, and enhances prediction accuracy. With the help of SILP, we only need to add one line of code to conventional existing algorithms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_11962 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Structured First-Layer Initialization Pre-Training Techniques to Accelerate Training Process Based on $\varepsilon$-Rank Tang, Tao Yang, Jiang Zhao, Yuxiang Zhu, Quanhui Numerical Analysis Training deep neural networks for scientific computing remains computationally expensive due to the slow formation of diverse feature representations in early training stages. Recent studies identify a staircase phenomenon in training dynamics, where loss decreases are closely correlated with increases in $\varepsilon$-rank, reflecting the effective number of linearly independent neuron functions. Motivated by this observation, this work proposes a structured first-layer initialization (SFLI) pre-training method to enhance the diversity of neural features at initialization by constructing $\varepsilon$-linearly independent neurons in the input layer. We present systematic initialization schemes compatible with various activation functions and integrate the strategy into multiple neural architectures, including modified multi-layer perceptrons and physics-informed residual adaptive networks. Extensive numerical experiments on function approximation and PDE benchmarks, demonstrate that SFLI significantly improves the initial $\varepsilon$-rank, accelerates convergence, mitigates spectral bias, and enhances prediction accuracy. With the help of SILP, we only need to add one line of code to conventional existing algorithms. |
| title | Structured First-Layer Initialization Pre-Training Techniques to Accelerate Training Process Based on $\varepsilon$-Rank |
| topic | Numerical Analysis |
| url | https://arxiv.org/abs/2507.11962 |