Self-Supervised Learning as Constrained Free-Energy Systems
Fuente:
Zenodo
Enregistré dans:
| Auteur principal: | |
|---|---|
| Format: | Recurso digital |
| Publié: |
Zenodo
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866901445422350336 |
|---|---|
| author | Kemple, Kurtis |
| author_facet | Kemple, Kurtis |
| contents | <p>This paper proposes that self-supervised learning methods are physical systems minimizing free energy under representational constraints, explaining why diverse approaches (VICReg, DINO, SimCLR, BYOL, Barlow Twins, JEPA) converge on similar hyperparameter ranges despite different theoretical motivations. The framework decomposes total free-energy deviation as F[q_t] - F* = κ + CD(t), where κ represents irreducible structural costs from architectural constraints and CD(t) measures dynamic misalignment. Training succeeds when gradient flow reduces CD(t) faster than constraints inflate κ. Organizational overhead η—the fraction of capacity consumed by coherence maintenance—must remain below a critical threshold η_c for stable representations. Documented empirical phenomena receive unified interpretation: variance-collapse universality (all SSL methods fail when embedding variance approaches zero); momentum convergence (BYOL, DINO, MoCo independently discover m ≈ 0.996, creating timescale τ = 1/(1-m) ≈ 250 steps matching characteristic relaxation times); batch size scaling (SimCLR requires ~4096 samples for manifold percolation); and depth thresholds (transformers exhibit emergent capabilities around 10-12 layers). These narrow ranges suggest underlying constraint boundaries. Each method implements the same physics through different mechanisms: VICReg's variance/covariance terms maintain dimensional spread; DINO's momentum creates timescale separation for stable reference tracking; SimCLR's negative samples ensure manifold coverage; BYOL's predictor breaks symmetry; Barlow Twins' decorrelation reduces redundancy; JEPA's prediction horizon enables recursive temporal coherence. All keep organizational overhead subcritical. The paper connects to the constraint eigenvalue framework's triplet architecture, proposing that SSL systems realize some eigenbranch configuration. Physical and biological systems typically follow the decagonal eigenbranch (π, φ, 10), but whether SSL matches this or discovers architecture-specific values remains an open question the framework helps sharpen. The convergence of independent research groups on similar thresholds suggests they discovered the same underlying constraint geometry through different optimization paths.</p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_18292515 |
| institution | Zenodo |
| language | |
| publishDate | 2026 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | Self-Supervised Learning as Constrained Free-Energy Systems Kemple, Kurtis self-supervised learning free energy principle representation learning neural networks thermodynamics DINO VICReg contrastive learning collapse prevention machine learning theory <p>This paper proposes that self-supervised learning methods are physical systems minimizing free energy under representational constraints, explaining why diverse approaches (VICReg, DINO, SimCLR, BYOL, Barlow Twins, JEPA) converge on similar hyperparameter ranges despite different theoretical motivations. The framework decomposes total free-energy deviation as F[q_t] - F* = κ + CD(t), where κ represents irreducible structural costs from architectural constraints and CD(t) measures dynamic misalignment. Training succeeds when gradient flow reduces CD(t) faster than constraints inflate κ. Organizational overhead η—the fraction of capacity consumed by coherence maintenance—must remain below a critical threshold η_c for stable representations. Documented empirical phenomena receive unified interpretation: variance-collapse universality (all SSL methods fail when embedding variance approaches zero); momentum convergence (BYOL, DINO, MoCo independently discover m ≈ 0.996, creating timescale τ = 1/(1-m) ≈ 250 steps matching characteristic relaxation times); batch size scaling (SimCLR requires ~4096 samples for manifold percolation); and depth thresholds (transformers exhibit emergent capabilities around 10-12 layers). These narrow ranges suggest underlying constraint boundaries. Each method implements the same physics through different mechanisms: VICReg's variance/covariance terms maintain dimensional spread; DINO's momentum creates timescale separation for stable reference tracking; SimCLR's negative samples ensure manifold coverage; BYOL's predictor breaks symmetry; Barlow Twins' decorrelation reduces redundancy; JEPA's prediction horizon enables recursive temporal coherence. All keep organizational overhead subcritical. The paper connects to the constraint eigenvalue framework's triplet architecture, proposing that SSL systems realize some eigenbranch configuration. Physical and biological systems typically follow the decagonal eigenbranch (π, φ, 10), but whether SSL matches this or discovers architecture-specific values remains an open question the framework helps sharpen. The convergence of independent research groups on similar thresholds suggests they discovered the same underlying constraint geometry through different optimization paths.</p> |
| title | Self-Supervised Learning as Constrained Free-Energy Systems |
| topic | self-supervised learning free energy principle representation learning neural networks thermodynamics DINO VICReg contrastive learning collapse prevention machine learning theory |
| url | https://doi.org/10.5281/zenodo.18292515 |