Mean-Field Limits for Two-Layer Neural Networks Trained with Consensus-Based Optimization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914176999358464 |
|---|---|
| author | De Deyn, William Herty, Michael Samaey, Giovanni |
| author_facet | De Deyn, William Herty, Michael Samaey, Giovanni |
| contents | We study Consensus-Based Optimization (CBO) for two-layer neural network training. We compare the performance of CBO against Adam on two test cases and demonstrate how a hybrid approach, combining CBO with Adam, provides faster convergence than CBO. Additionally, in the context of multi-task learning, we recast CBO into a formulation that offers less memory overhead. The CBO method allows for a mean-field limit formulation, which we couple with the mean-field limit of the neural network. To this end, we first reformulate CBO within the optimal transport framework. In the limit of infinitely many particles, we define the corresponding dynamics on the Wasserstein-over-Wasserstein space and show that the variance decreases monotonically. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_21466 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Mean-Field Limits for Two-Layer Neural Networks Trained with Consensus-Based Optimization De Deyn, William Herty, Michael Samaey, Giovanni Machine Learning Optimization and Control We study Consensus-Based Optimization (CBO) for two-layer neural network training. We compare the performance of CBO against Adam on two test cases and demonstrate how a hybrid approach, combining CBO with Adam, provides faster convergence than CBO. Additionally, in the context of multi-task learning, we recast CBO into a formulation that offers less memory overhead. The CBO method allows for a mean-field limit formulation, which we couple with the mean-field limit of the neural network. To this end, we first reformulate CBO within the optimal transport framework. In the limit of infinitely many particles, we define the corresponding dynamics on the Wasserstein-over-Wasserstein space and show that the variance decreases monotonically. |
| title | Mean-Field Limits for Two-Layer Neural Networks Trained with Consensus-Based Optimization |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2511.21466 |