Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.18452 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912369356046336 |
|---|---|
| author | Weng, Xi An, Jianing Ma, Xudong Qi, Binhang Luo, Jie Yang, Xi Dong, Jin Song Huang, Lei |
| author_facet | Weng, Xi An, Jianing Ma, Xudong Qi, Binhang Luo, Jie Yang, Xi Dong, Jin Song Huang, Lei |
| contents | Self-supervised learning (SSL) methods via joint embedding architectures have proven remarkably effective at capturing semantically rich representations with strong clustering properties, magically in the absence of label supervision. Despite this, few of them have explored leveraging these untapped properties to improve themselves. In this paper, we provide an evidence through various metrics that the encoder's output $encoding$ exhibits superior and more stable clustering properties compared to other components. Building on this insight, we propose a novel positive-feedback SSL method, termed Representation Self-Assignment (ReSA), which leverages the model's clustering properties to promote learning in a self-guided manner. Extensive experiments on standard SSL benchmarks reveal that models pretrained with ReSA outperform other state-of-the-art SSL methods by a significant margin. Finally, we analyze how ReSA facilitates better clustering properties, demonstrating that it effectively enhances clustering performance at both fine-grained and coarse-grained levels, shaping representations that are inherently more structured and semantically meaningful. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_18452 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Clustering Properties of Self-Supervised Learning Weng, Xi An, Jianing Ma, Xudong Qi, Binhang Luo, Jie Yang, Xi Dong, Jin Song Huang, Lei Machine Learning Artificial Intelligence Self-supervised learning (SSL) methods via joint embedding architectures have proven remarkably effective at capturing semantically rich representations with strong clustering properties, magically in the absence of label supervision. Despite this, few of them have explored leveraging these untapped properties to improve themselves. In this paper, we provide an evidence through various metrics that the encoder's output $encoding$ exhibits superior and more stable clustering properties compared to other components. Building on this insight, we propose a novel positive-feedback SSL method, termed Representation Self-Assignment (ReSA), which leverages the model's clustering properties to promote learning in a self-guided manner. Extensive experiments on standard SSL benchmarks reveal that models pretrained with ReSA outperform other state-of-the-art SSL methods by a significant margin. Finally, we analyze how ReSA facilitates better clustering properties, demonstrating that it effectively enhances clustering performance at both fine-grained and coarse-grained levels, shaping representations that are inherently more structured and semantically meaningful. |
| title | Clustering Properties of Self-Supervised Learning |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2501.18452 |