Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.10812 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913133083230208 |
|---|---|
| author | Calder, Jeff Lee, Wonjun |
| author_facet | Calder, Jeff Lee, Wonjun |
| contents | Contrastive learning effectively clusters data despite a loss landscape filled with poor solutions, a success that is heavily dependent on the choice of data augmentations. How optimization consistently finds meaningful patterns remains an open question. We show this success stems from training dynamics rather than the loss function alone. Crucially, under a highly specific structural assumption governing the connectivity and variance of the data augmentations, we prove that once a critical spectral alignment threshold is reached, data features inevitably and rapidly separate into distinct clusters. We establish this mechanism for both discrete datasets and the macroscopic continuum limit, modeling latent dynamics as a Wasserstein gradient flow to demonstrate that this separation persists as the number of data points approaches infinity. We hypothesize that natural training dynamics inherently drive the system toward this critical state. We extensively validate this empirically across four diverse domains (synthetic shapes, images, text, and PDEs). In every setting, a sharp increase in this spectral quantity consistently precedes clean data separation, revealing that contrastive learning's success is governed by a dynamically emerging trigger tightly coupled to the underlying augmentation structure. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_10812 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Dynamics Over Landscape: The Emergence of Linear Separability via Spectral Alignment in Contrastive Learning Calder, Jeff Lee, Wonjun Numerical Analysis Contrastive learning effectively clusters data despite a loss landscape filled with poor solutions, a success that is heavily dependent on the choice of data augmentations. How optimization consistently finds meaningful patterns remains an open question. We show this success stems from training dynamics rather than the loss function alone. Crucially, under a highly specific structural assumption governing the connectivity and variance of the data augmentations, we prove that once a critical spectral alignment threshold is reached, data features inevitably and rapidly separate into distinct clusters. We establish this mechanism for both discrete datasets and the macroscopic continuum limit, modeling latent dynamics as a Wasserstein gradient flow to demonstrate that this separation persists as the number of data points approaches infinity. We hypothesize that natural training dynamics inherently drive the system toward this critical state. We extensively validate this empirically across four diverse domains (synthetic shapes, images, text, and PDEs). In every setting, a sharp increase in this spectral quantity consistently precedes clean data separation, revealing that contrastive learning's success is governed by a dynamically emerging trigger tightly coupled to the underlying augmentation structure. |
| title | Dynamics Over Landscape: The Emergence of Linear Separability via Spectral Alignment in Contrastive Learning |
| topic | Numerical Analysis |
| url | https://arxiv.org/abs/2503.10812 |