Modulate Your Spectrum in Self-Supervised Learning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911761399021568 |
|---|---|
| author | Weng, Xi Ni, Yunhao Song, Tengwei Luo, Jie Anwer, Rao Muhammad Khan, Salman Khan, Fahad Shahbaz Huang, Lei |
| author_facet | Weng, Xi Ni, Yunhao Song, Tengwei Luo, Jie Anwer, Rao Muhammad Khan, Salman Khan, Fahad Shahbaz Huang, Lei |
| contents | Whitening loss offers a theoretical guarantee against feature collapse in self-supervised learning (SSL) with joint embedding architectures. Typically, it involves a hard whitening approach, transforming the embedding and applying loss to the whitened output. In this work, we introduce Spectral Transformation (ST), a framework to modulate the spectrum of embedding and to seek for functions beyond whitening that can avoid dimensional collapse. We show that whitening is a special instance of ST by definition, and our empirical investigations unveil other ST instances capable of preventing collapse. Additionally, we propose a novel ST instance named IterNorm with trace loss (INTL). Theoretical analysis confirms INTL's efficacy in preventing collapse and modulating the spectrum of embedding toward equal-eigenvalues during optimization. Our experiments on ImageNet classification and COCO object detection demonstrate INTL's potential in learning superior representations. The code is available at https://github.com/winci-ai/INTL. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_16789 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Modulate Your Spectrum in Self-Supervised Learning Weng, Xi Ni, Yunhao Song, Tengwei Luo, Jie Anwer, Rao Muhammad Khan, Salman Khan, Fahad Shahbaz Huang, Lei Machine Learning Computer Vision and Pattern Recognition Signal Processing Whitening loss offers a theoretical guarantee against feature collapse in self-supervised learning (SSL) with joint embedding architectures. Typically, it involves a hard whitening approach, transforming the embedding and applying loss to the whitened output. In this work, we introduce Spectral Transformation (ST), a framework to modulate the spectrum of embedding and to seek for functions beyond whitening that can avoid dimensional collapse. We show that whitening is a special instance of ST by definition, and our empirical investigations unveil other ST instances capable of preventing collapse. Additionally, we propose a novel ST instance named IterNorm with trace loss (INTL). Theoretical analysis confirms INTL's efficacy in preventing collapse and modulating the spectrum of embedding toward equal-eigenvalues during optimization. Our experiments on ImageNet classification and COCO object detection demonstrate INTL's potential in learning superior representations. The code is available at https://github.com/winci-ai/INTL. |
| title | Modulate Your Spectrum in Self-Supervised Learning |
| topic | Machine Learning Computer Vision and Pattern Recognition Signal Processing |
| url | https://arxiv.org/abs/2305.16789 |