Self-supervised Representation Learning From Random Data Projectors
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917618746654720 |
|---|---|
| author | Sui, Yi Wu, Tongzi Cresswell, Jesse C. Wu, Ga Stein, George Huang, Xiao Shi Zhang, Xiaochen Volkovs, Maksims |
| author_facet | Sui, Yi Wu, Tongzi Cresswell, Jesse C. Wu, Ga Stein, George Huang, Xiao Shi Zhang, Xiaochen Volkovs, Maksims |
| contents | Self-supervised representation learning~(SSRL) has advanced considerably by exploiting the transformation invariance assumption under artificially designed data augmentations. While augmentation-based SSRL algorithms push the boundaries of performance in computer vision and natural language processing, they are often not directly applicable to other data modalities, and can conflict with application-specific data augmentation constraints. This paper presents an SSRL approach that can be applied to any data modality and network architecture because it does not rely on augmentations or masking. Specifically, we show that high-quality data representations can be learned by reconstructing random data projections. We evaluate the proposed approach on a wide range of representation learning tasks that span diverse modalities and real-world applications. We show that it outperforms multiple state-of-the-art SSRL baselines. Due to its wide applicability and strong empirical results, we argue that learning from randomness is a fruitful research direction worthy of attention and further study. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_07756 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Self-supervised Representation Learning From Random Data Projectors Sui, Yi Wu, Tongzi Cresswell, Jesse C. Wu, Ga Stein, George Huang, Xiao Shi Zhang, Xiaochen Volkovs, Maksims Machine Learning Self-supervised representation learning~(SSRL) has advanced considerably by exploiting the transformation invariance assumption under artificially designed data augmentations. While augmentation-based SSRL algorithms push the boundaries of performance in computer vision and natural language processing, they are often not directly applicable to other data modalities, and can conflict with application-specific data augmentation constraints. This paper presents an SSRL approach that can be applied to any data modality and network architecture because it does not rely on augmentations or masking. Specifically, we show that high-quality data representations can be learned by reconstructing random data projections. We evaluate the proposed approach on a wide range of representation learning tasks that span diverse modalities and real-world applications. We show that it outperforms multiple state-of-the-art SSRL baselines. Due to its wide applicability and strong empirical results, we argue that learning from randomness is a fruitful research direction worthy of attention and further study. |
| title | Self-supervised Representation Learning From Random Data Projectors |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2310.07756 |