MixDiff: Mixing Natural and Synthetic Images for Robust Self-Supervised Representations
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910727688683520 |
|---|---|
| author | Bafghi, Reza Akbarian Harilal, Nidhin Monteleoni, Claire Raissi, Maziar |
| author_facet | Bafghi, Reza Akbarian Harilal, Nidhin Monteleoni, Claire Raissi, Maziar |
| contents | This paper introduces MixDiff, a new self-supervised learning (SSL) pre-training framework that combines real and synthetic images. Unlike traditional SSL methods that predominantly use real images, MixDiff uses a variant of Stable Diffusion to replace an augmented instance of a real image, facilitating the learning of cross real-synthetic image representations. Our key insight is that while models trained solely on synthetic images underperform, combining real and synthetic data leads to more robust and adaptable representations. Experiments show MixDiff enhances SimCLR, BarlowTwins, and DINO across various robustness datasets and domain transfer tasks, boosting SimCLR's ImageNet-1K accuracy by 4.56%. Our framework also demonstrates comparable performance without needing any augmentations, a surprising finding in SSL where augmentations are typically crucial. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_12368 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MixDiff: Mixing Natural and Synthetic Images for Robust Self-Supervised Representations Bafghi, Reza Akbarian Harilal, Nidhin Monteleoni, Claire Raissi, Maziar Computer Vision and Pattern Recognition This paper introduces MixDiff, a new self-supervised learning (SSL) pre-training framework that combines real and synthetic images. Unlike traditional SSL methods that predominantly use real images, MixDiff uses a variant of Stable Diffusion to replace an augmented instance of a real image, facilitating the learning of cross real-synthetic image representations. Our key insight is that while models trained solely on synthetic images underperform, combining real and synthetic data leads to more robust and adaptable representations. Experiments show MixDiff enhances SimCLR, BarlowTwins, and DINO across various robustness datasets and domain transfer tasks, boosting SimCLR's ImageNet-1K accuracy by 4.56%. Our framework also demonstrates comparable performance without needing any augmentations, a surprising finding in SSL where augmentations are typically crucial. |
| title | MixDiff: Mixing Natural and Synthetic Images for Robust Self-Supervised Representations |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.12368 |