The Price of Implicit Bias in Adversarially Robust Generalization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909219306864640 |
|---|---|
| author | Tsilivis, Nikolaos Frank, Natalie Srebro, Nathan Kempe, Julia |
| author_facet | Tsilivis, Nikolaos Frank, Natalie Srebro, Nathan Kempe, Julia |
| contents | We study the implicit bias of optimization in robust empirical risk minimization (robust ERM) and its connection with robust generalization. In classification settings under adversarial perturbations with linear models, we study what type of regularization should ideally be applied for a given perturbation set to improve (robust) generalization. We then show that the implicit bias of optimization in robust ERM can significantly affect the robustness of the model and identify two ways this can happen; either through the optimization algorithm or the architecture. We verify our predictions in simulations with synthetic data and experimentally study the importance of implicit bias in robust ERM with deep neural networks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_04981 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Price of Implicit Bias in Adversarially Robust Generalization Tsilivis, Nikolaos Frank, Natalie Srebro, Nathan Kempe, Julia Machine Learning We study the implicit bias of optimization in robust empirical risk minimization (robust ERM) and its connection with robust generalization. In classification settings under adversarial perturbations with linear models, we study what type of regularization should ideally be applied for a given perturbation set to improve (robust) generalization. We then show that the implicit bias of optimization in robust ERM can significantly affect the robustness of the model and identify two ways this can happen; either through the optimization algorithm or the architecture. We verify our predictions in simulations with synthetic data and experimentally study the importance of implicit bias in robust ERM with deep neural networks. |
| title | The Price of Implicit Bias in Adversarially Robust Generalization |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2406.04981 |