On the Convergence of Overparameterized Problems: Inherent Properties of the Compositional Structure of Neural Networks
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908649004204032 |
|---|---|
| author | de Oliveira, Arthur Castello Branco Jatkar, Dhruv Sontag, Eduardo |
| author_facet | de Oliveira, Arthur Castello Branco Jatkar, Dhruv Sontag, Eduardo |
| contents | This paper investigates how the compositional structure of neural networks shapes their optimization landscape and training dynamics. We analyze the gradient flow associated with overparameterized optimization problems, which can be interpreted as training a neural network with linear activations. Remarkably, we show that the global convergence properties can be derived for any cost function that is proper and real analytic. We then specialize the analysis to scalar-valued cost functions, where the geometry of the landscape can be fully characterized. In this setting, we demonstrate that key structural features -- such as the location and stability of saddle points -- are universal across all admissible costs, depending solely on the overparameterized representation rather than on problem-specific details. Moreover, we show that convergence can be arbitrarily accelerated depending on the initialization, as measured by an imbalance metric introduced in this work. Finally, we discuss how these insights may generalize to neural networks with sigmoidal activations, showing through a simple example which geometric and dynamical properties persist beyond the linear case. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_09810 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | On the Convergence of Overparameterized Problems: Inherent Properties of the Compositional Structure of Neural Networks de Oliveira, Arthur Castello Branco Jatkar, Dhruv Sontag, Eduardo Machine Learning Artificial Intelligence Systems and Control Optimization and Control This paper investigates how the compositional structure of neural networks shapes their optimization landscape and training dynamics. We analyze the gradient flow associated with overparameterized optimization problems, which can be interpreted as training a neural network with linear activations. Remarkably, we show that the global convergence properties can be derived for any cost function that is proper and real analytic. We then specialize the analysis to scalar-valued cost functions, where the geometry of the landscape can be fully characterized. In this setting, we demonstrate that key structural features -- such as the location and stability of saddle points -- are universal across all admissible costs, depending solely on the overparameterized representation rather than on problem-specific details. Moreover, we show that convergence can be arbitrarily accelerated depending on the initialization, as measured by an imbalance metric introduced in this work. Finally, we discuss how these insights may generalize to neural networks with sigmoidal activations, showing through a simple example which geometric and dynamical properties persist beyond the linear case. |
| title | On the Convergence of Overparameterized Problems: Inherent Properties of the Compositional Structure of Neural Networks |
| topic | Machine Learning Artificial Intelligence Systems and Control Optimization and Control |
| url | https://arxiv.org/abs/2511.09810 |