On the Geometry of Regularization in Adversarial Training: High-Dimensional Asymptotics and Generalization Bounds
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909357749305344 |
|---|---|
| author | Vilucchio, Matteo Tsilivis, Nikolaos Loureiro, Bruno Kempe, Julia |
| author_facet | Vilucchio, Matteo Tsilivis, Nikolaos Loureiro, Bruno Kempe, Julia |
| contents | Regularization, whether explicit in terms of a penalty in the loss or implicit in the choice of algorithm, is a cornerstone of modern machine learning. Indeed, controlling the complexity of the model class is particularly important when data is scarce, noisy or contaminated, as it translates a statistical belief on the underlying structure of the data. This work investigates the question of how to choose the regularization norm $\lVert \cdot \rVert$ in the context of high-dimensional adversarial training for binary classification. To this end, we first derive an exact asymptotic description of the robust, regularized empirical risk minimizer for various types of adversarial attacks and regularization norms (including non-$\ell_p$ norms). We complement this analysis with a uniform convergence analysis, deriving bounds on the Rademacher Complexity for this class of problems. Leveraging our theoretical results, we quantitatively characterize the relationship between perturbation size and the optimal choice of $\lVert \cdot \rVert$, confirming the intuition that, in the data scarce regime, the type of regularization becomes increasingly important for adversarial training as perturbations grow in size. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_16073 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On the Geometry of Regularization in Adversarial Training: High-Dimensional Asymptotics and Generalization Bounds Vilucchio, Matteo Tsilivis, Nikolaos Loureiro, Bruno Kempe, Julia Machine Learning Disordered Systems and Neural Networks Statistics Theory Regularization, whether explicit in terms of a penalty in the loss or implicit in the choice of algorithm, is a cornerstone of modern machine learning. Indeed, controlling the complexity of the model class is particularly important when data is scarce, noisy or contaminated, as it translates a statistical belief on the underlying structure of the data. This work investigates the question of how to choose the regularization norm $\lVert \cdot \rVert$ in the context of high-dimensional adversarial training for binary classification. To this end, we first derive an exact asymptotic description of the robust, regularized empirical risk minimizer for various types of adversarial attacks and regularization norms (including non-$\ell_p$ norms). We complement this analysis with a uniform convergence analysis, deriving bounds on the Rademacher Complexity for this class of problems. Leveraging our theoretical results, we quantitatively characterize the relationship between perturbation size and the optimal choice of $\lVert \cdot \rVert$, confirming the intuition that, in the data scarce regime, the type of regularization becomes increasingly important for adversarial training as perturbations grow in size. |
| title | On the Geometry of Regularization in Adversarial Training: High-Dimensional Asymptotics and Generalization Bounds |
| topic | Machine Learning Disordered Systems and Neural Networks Statistics Theory |
| url | https://arxiv.org/abs/2410.16073 |