Randomly Initialized Networks Can Learn from Peer-to-Peer Consensus
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909002898604032 |
|---|---|
| author | Rodríguez-Betancourt, Esteban Casasola-Murillo, Edgar |
| author_facet | Rodríguez-Betancourt, Esteban Casasola-Murillo, Edgar |
| contents | In self-supervised learning, self-distilled methods have shown impressive performance, learning representations useful for downstream tasks and even displaying emergent properties. However, state-of-the-art methods usually rely on ensembles of complex mechanisms, with many design choices that are empirically motivated and not well understood.
In this work, we explore the role of self-distillation within learning dynamics. Specifically, we isolate the effect of self-distillation by training a group of randomly initialized networks, removing all other common components such as projectors, predictors, and even pretext tasks. Our findings show that even this minimal setup can lead to learned representations with non-trivial improvements over a random baseline on downstream tasks. We also demonstrate how this effect varies with different hyperparameters and present a short analysis of what is being learned by the models under this setup. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_18390 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Randomly Initialized Networks Can Learn from Peer-to-Peer Consensus Rodríguez-Betancourt, Esteban Casasola-Murillo, Edgar Machine Learning Artificial Intelligence In self-supervised learning, self-distilled methods have shown impressive performance, learning representations useful for downstream tasks and even displaying emergent properties. However, state-of-the-art methods usually rely on ensembles of complex mechanisms, with many design choices that are empirically motivated and not well understood. In this work, we explore the role of self-distillation within learning dynamics. Specifically, we isolate the effect of self-distillation by training a group of randomly initialized networks, removing all other common components such as projectors, predictors, and even pretext tasks. Our findings show that even this minimal setup can lead to learned representations with non-trivial improvements over a random baseline on downstream tasks. We also demonstrate how this effect varies with different hyperparameters and present a short analysis of what is being learned by the models under this setup. |
| title | Randomly Initialized Networks Can Learn from Peer-to-Peer Consensus |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2604.18390 |