Criticality Leveraged Adversarial Training (CLAT) for Boosted Performance via Parameter Efficiency
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908170645929984 |
|---|---|
| author | Gopal, Bhavna Yang, Huanrui Zhang, Jingyang Horton, Mark Chen, Yiran |
| author_facet | Gopal, Bhavna Yang, Huanrui Zhang, Jingyang Horton, Mark Chen, Yiran |
| contents | Adversarial training enhances neural network robustness but suffers from a tendency to overfit and increased generalization errors on clean data. This work introduces CLAT, an innovative approach that mitigates adversarial overfitting by introducing parameter efficiency into the adversarial training process, improving both clean accuracy and adversarial robustness. Instead of tuning the entire model, CLAT identifies and fine-tunes robustness-critical layers - those predominantly learning non-robust features - while freezing the remaining model to enhance robustness. It employs dynamic critical layer selection to adapt to changes in layer criticality throughout the fine-tuning process. Empirically, CLAT can be applied on top of existing adversarial training methods, significantly reduces the number of trainable parameters by approximately 95%, and achieves more than a 2% improvement in adversarial robustness compared to baseline methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_10204 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Criticality Leveraged Adversarial Training (CLAT) for Boosted Performance via Parameter Efficiency Gopal, Bhavna Yang, Huanrui Zhang, Jingyang Horton, Mark Chen, Yiran Machine Learning Computer Vision and Pattern Recognition Adversarial training enhances neural network robustness but suffers from a tendency to overfit and increased generalization errors on clean data. This work introduces CLAT, an innovative approach that mitigates adversarial overfitting by introducing parameter efficiency into the adversarial training process, improving both clean accuracy and adversarial robustness. Instead of tuning the entire model, CLAT identifies and fine-tunes robustness-critical layers - those predominantly learning non-robust features - while freezing the remaining model to enhance robustness. It employs dynamic critical layer selection to adapt to changes in layer criticality throughout the fine-tuning process. Empirically, CLAT can be applied on top of existing adversarial training methods, significantly reduces the number of trainable parameters by approximately 95%, and achieves more than a 2% improvement in adversarial robustness compared to baseline methods. |
| title | Criticality Leveraged Adversarial Training (CLAT) for Boosted Performance via Parameter Efficiency |
| topic | Machine Learning Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.10204 |