An analysis of optimization problems involving ReLU neural networks
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929698842345472 |
|---|---|
| author | Plate, Christoph Hahn, Mirko Klimek, Alexander Ganzer, Caroline Sundmacher, Kai Sager, Sebastian |
| author_facet | Plate, Christoph Hahn, Mirko Klimek, Alexander Ganzer, Caroline Sundmacher, Kai Sager, Sebastian |
| contents | Solving mixed-integer optimization problems with embedded neural networks with ReLU activation functions is challenging. Big-M coefficients that arise in relaxing binary decisions related to these functions grow exponentially with the number of layers. We survey and propose different approaches to analyze and improve the run time behavior of mixed-integer programming solvers in this context. Among them are clipped variants and regularization techniques applied during training as well as optimization-based bound tightening and a novel scaling for given ReLU networks. We numerically compare these approaches for three benchmark problems from the literature. We use the number of linear regions, the percentage of stable neurons, and overall computational effort as indicators. As a major takeaway we observe and quantify a trade-off between the often desired redundancy of neural network models versus the computational costs for solving related optimization problems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_03016 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | An analysis of optimization problems involving ReLU neural networks Plate, Christoph Hahn, Mirko Klimek, Alexander Ganzer, Caroline Sundmacher, Kai Sager, Sebastian Optimization and Control Machine Learning Solving mixed-integer optimization problems with embedded neural networks with ReLU activation functions is challenging. Big-M coefficients that arise in relaxing binary decisions related to these functions grow exponentially with the number of layers. We survey and propose different approaches to analyze and improve the run time behavior of mixed-integer programming solvers in this context. Among them are clipped variants and regularization techniques applied during training as well as optimization-based bound tightening and a novel scaling for given ReLU networks. We numerically compare these approaches for three benchmark problems from the literature. We use the number of linear regions, the percentage of stable neurons, and overall computational effort as indicators. As a major takeaway we observe and quantify a trade-off between the often desired redundancy of neural network models versus the computational costs for solving related optimization problems. |
| title | An analysis of optimization problems involving ReLU neural networks |
| topic | Optimization and Control Machine Learning |
| url | https://arxiv.org/abs/2502.03016 |