The Hidden Pitfalls of the Cosine Similarity Loss
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916298798137344 |
|---|---|
| author | Draganov, Andrew Vadgama, Sharvaree Bekkers, Erik J. |
| author_facet | Draganov, Andrew Vadgama, Sharvaree Bekkers, Erik J. |
| contents | We show that the gradient of the cosine similarity between two points goes to zero in two under-explored settings: (1) if a point has large magnitude or (2) if the points are on opposite ends of the latent space. Counterintuitively, we prove that optimizing the cosine similarity between points forces them to grow in magnitude. Thus, (1) is unavoidable in practice. We then observe that these derivations are extremely general -- they hold across deep learning architectures and for many of the standard self-supervised learning (SSL) loss functions. This leads us to propose cut-initialization: a simple change to network initialization that helps all studied SSL methods converge faster. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_16468 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Hidden Pitfalls of the Cosine Similarity Loss Draganov, Andrew Vadgama, Sharvaree Bekkers, Erik J. Machine Learning We show that the gradient of the cosine similarity between two points goes to zero in two under-explored settings: (1) if a point has large magnitude or (2) if the points are on opposite ends of the latent space. Counterintuitively, we prove that optimizing the cosine similarity between points forces them to grow in magnitude. Thus, (1) is unavoidable in practice. We then observe that these derivations are extremely general -- they hold across deep learning architectures and for many of the standard self-supervised learning (SSL) loss functions. This leads us to propose cut-initialization: a simple change to network initialization that helps all studied SSL methods converge faster. |
| title | The Hidden Pitfalls of the Cosine Similarity Loss |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2406.16468 |