Fine-Tuning Text-To-Image Diffusion Models for Class-Wise Spurious Feature Generation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913233516888064 |
|---|---|
| author | MaungMaung, AprilPyone Nguyen, Huy H. Kiya, Hitoshi Echizen, Isao |
| author_facet | MaungMaung, AprilPyone Nguyen, Huy H. Kiya, Hitoshi Echizen, Isao |
| contents | We propose a method for generating spurious features by leveraging large-scale text-to-image diffusion models. Although the previous work detects spurious features in a large-scale dataset like ImageNet and introduces Spurious ImageNet, we found that not all spurious images are spurious across different classifiers. Although spurious images help measure the reliance of a classifier, filtering many images from the Internet to find more spurious features is time-consuming. To this end, we utilize an existing approach of personalizing large-scale text-to-image diffusion models with available discovered spurious images and propose a new spurious feature similarity loss based on neural features of an adversarially robust model. Precisely, we fine-tune Stable Diffusion with several reference images from Spurious ImageNet with a modified objective incorporating the proposed spurious-feature similarity loss. Experiment results show that our method can generate spurious images that are consistently spurious across different classifiers. Moreover, the generated spurious images are visually similar to reference images from Spurious ImageNet. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_08200 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Fine-Tuning Text-To-Image Diffusion Models for Class-Wise Spurious Feature Generation MaungMaung, AprilPyone Nguyen, Huy H. Kiya, Hitoshi Echizen, Isao Computer Vision and Pattern Recognition We propose a method for generating spurious features by leveraging large-scale text-to-image diffusion models. Although the previous work detects spurious features in a large-scale dataset like ImageNet and introduces Spurious ImageNet, we found that not all spurious images are spurious across different classifiers. Although spurious images help measure the reliance of a classifier, filtering many images from the Internet to find more spurious features is time-consuming. To this end, we utilize an existing approach of personalizing large-scale text-to-image diffusion models with available discovered spurious images and propose a new spurious feature similarity loss based on neural features of an adversarially robust model. Precisely, we fine-tune Stable Diffusion with several reference images from Spurious ImageNet with a modified objective incorporating the proposed spurious-feature similarity loss. Experiment results show that our method can generate spurious images that are consistently spurious across different classifiers. Moreover, the generated spurious images are visually similar to reference images from Spurious ImageNet. |
| title | Fine-Tuning Text-To-Image Diffusion Models for Class-Wise Spurious Feature Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2402.08200 |