Benchmarking Robust Self-Supervised Learning Across Diverse Downstream Tasks
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916329203695616 |
|---|---|
| author | Kowalczuk, Antoni Dubiński, Jan Ghomi, Atiyeh Ashari Sui, Yi Stein, George Wu, Jiapeng Cresswell, Jesse C. Boenisch, Franziska Dziedzic, Adam |
| author_facet | Kowalczuk, Antoni Dubiński, Jan Ghomi, Atiyeh Ashari Sui, Yi Stein, George Wu, Jiapeng Cresswell, Jesse C. Boenisch, Franziska Dziedzic, Adam |
| contents | Large-scale vision models have become integral in many applications due to their unprecedented performance and versatility across downstream tasks. However, the robustness of these foundation models has primarily been explored for a single task, namely image classification. The vulnerability of other common vision tasks, such as semantic segmentation and depth estimation, remains largely unknown. We present a comprehensive empirical evaluation of the adversarial robustness of self-supervised vision encoders across multiple downstream tasks. Our attacks operate in the encoder embedding space and at the downstream task output level. In both cases, current state-of-the-art adversarial fine-tuning techniques tested only for classification significantly degrade clean and robust performance on other tasks. Since the purpose of a foundation model is to cater to multiple applications at once, our findings reveal the need to enhance encoder robustness more broadly. Our code is available at ${github.com/layer6ai-labs/ssl-robustness}$. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_12588 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Benchmarking Robust Self-Supervised Learning Across Diverse Downstream Tasks Kowalczuk, Antoni Dubiński, Jan Ghomi, Atiyeh Ashari Sui, Yi Stein, George Wu, Jiapeng Cresswell, Jesse C. Boenisch, Franziska Dziedzic, Adam Computer Vision and Pattern Recognition Artificial Intelligence Large-scale vision models have become integral in many applications due to their unprecedented performance and versatility across downstream tasks. However, the robustness of these foundation models has primarily been explored for a single task, namely image classification. The vulnerability of other common vision tasks, such as semantic segmentation and depth estimation, remains largely unknown. We present a comprehensive empirical evaluation of the adversarial robustness of self-supervised vision encoders across multiple downstream tasks. Our attacks operate in the encoder embedding space and at the downstream task output level. In both cases, current state-of-the-art adversarial fine-tuning techniques tested only for classification significantly degrade clean and robust performance on other tasks. Since the purpose of a foundation model is to cater to multiple applications at once, our findings reveal the need to enhance encoder robustness more broadly. Our code is available at ${github.com/layer6ai-labs/ssl-robustness}$. |
| title | Benchmarking Robust Self-Supervised Learning Across Diverse Downstream Tasks |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2407.12588 |