Evaluating the Evaluators: Trust in Adversarial Robustness Tests
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915371868487680 |
|---|---|
| author | Cinà, Antonio Emanuele Pintor, Maura Demetrio, Luca Demontis, Ambra Biggio, Battista Roli, Fabio |
| author_facet | Cinà, Antonio Emanuele Pintor, Maura Demetrio, Luca Demontis, Ambra Biggio, Battista Roli, Fabio |
| contents | Despite significant progress in designing powerful adversarial evasion attacks for robustness verification, the evaluation of these methods often remains inconsistent and unreliable. Many assessments rely on mismatched models, unverified implementations, and uneven computational budgets, which can lead to biased results and a false sense of security. Consequently, robustness claims built on such flawed testing protocols may be misleading and give a false sense of security. As a concrete step toward improving evaluation reliability, we present AttackBench, a benchmark framework developed to assess the effectiveness of gradient-based attacks under standardized and reproducible conditions. AttackBench serves as an evaluation tool that ranks existing attack implementations based on a novel optimality metric, which enables researchers and practitioners to identify the most reliable and effective attack for use in subsequent robustness evaluations. The framework enforces consistent testing conditions and enables continuous updates, making it a reliable foundation for robustness verification. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_03450 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Evaluating the Evaluators: Trust in Adversarial Robustness Tests Cinà, Antonio Emanuele Pintor, Maura Demetrio, Luca Demontis, Ambra Biggio, Battista Roli, Fabio Cryptography and Security Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Despite significant progress in designing powerful adversarial evasion attacks for robustness verification, the evaluation of these methods often remains inconsistent and unreliable. Many assessments rely on mismatched models, unverified implementations, and uneven computational budgets, which can lead to biased results and a false sense of security. Consequently, robustness claims built on such flawed testing protocols may be misleading and give a false sense of security. As a concrete step toward improving evaluation reliability, we present AttackBench, a benchmark framework developed to assess the effectiveness of gradient-based attacks under standardized and reproducible conditions. AttackBench serves as an evaluation tool that ranks existing attack implementations based on a novel optimality metric, which enables researchers and practitioners to identify the most reliable and effective attack for use in subsequent robustness evaluations. The framework enforces consistent testing conditions and enables continuous updates, making it a reliable foundation for robustness verification. |
| title | Evaluating the Evaluators: Trust in Adversarial Robustness Tests |
| topic | Cryptography and Security Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2507.03450 |