A Layer Separation Optimization Framework for Cross-Entropy Training in Deep Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915958241624064 |
|---|---|
| author | Liu, Yaru Ng, Michael K. Gu, Yiqi |
| author_facet | Liu, Yaru Ng, Michael K. Gu, Yiqi |
| contents | This paper investigates the deep learning optimization problem with softmax cross-entropy loss. We propose a layer separation strategy to alleviate the strong nonconvexity encountered during training deep networks. For cross-entropy models with fully connected and convolutional neural networks, we introduce auxiliary variables associated with hidden layer outputs and construct corresponding layer separation models, which decompose the original deeply nested optimization problem into a sequence of more manageable subproblems. We also conduct theoretical analyses, proving that the new layer separation loss provides an upper bound for the original cross-entropy loss. Moreover, we design alternating minimization algorithms and prove that, under appropriate conditions, these algorithms exhibit decreasing properties of the loss function. Numerical experiments validate the effectiveness of the proposed methods and indicate improved optimization behavior, especially for fully connected and convolutional neural networks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_23225 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | A Layer Separation Optimization Framework for Cross-Entropy Training in Deep Learning Liu, Yaru Ng, Michael K. Gu, Yiqi Machine Learning Optimization and Control 65K10, 68T07, 90C30 This paper investigates the deep learning optimization problem with softmax cross-entropy loss. We propose a layer separation strategy to alleviate the strong nonconvexity encountered during training deep networks. For cross-entropy models with fully connected and convolutional neural networks, we introduce auxiliary variables associated with hidden layer outputs and construct corresponding layer separation models, which decompose the original deeply nested optimization problem into a sequence of more manageable subproblems. We also conduct theoretical analyses, proving that the new layer separation loss provides an upper bound for the original cross-entropy loss. Moreover, we design alternating minimization algorithms and prove that, under appropriate conditions, these algorithms exhibit decreasing properties of the loss function. Numerical experiments validate the effectiveness of the proposed methods and indicate improved optimization behavior, especially for fully connected and convolutional neural networks. |
| title | A Layer Separation Optimization Framework for Cross-Entropy Training in Deep Learning |
| topic | Machine Learning Optimization and Control 65K10, 68T07, 90C30 |
| url | https://arxiv.org/abs/2604.23225 |