Poly-View Contrastive Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929268762607616 |
|---|---|
| author | Shidani, Amitis Hjelm, Devon Ramapuram, Jason Webb, Russ Dhekane, Eeshan Gunesh Busbridge, Dan |
| author_facet | Shidani, Amitis Hjelm, Devon Ramapuram, Jason Webb, Russ Dhekane, Eeshan Gunesh Busbridge, Dan |
| contents | Contrastive learning typically matches pairs of related views among a number of unrelated negative views. Views can be generated (e.g. by augmentations) or be observed. We investigate matching when there are more than two related views which we call poly-view tasks, and derive new representation learning objectives using information maximization and sufficient statistics. We show that with unlimited computation, one should maximize the number of related views, and with a fixed compute budget, it is beneficial to decrease the number of unique samples whilst increasing the number of views of those samples. In particular, poly-view contrastive models trained for 128 epochs with batch size 256 outperform SimCLR trained for 1024 epochs at batch size 4096 on ImageNet1k, challenging the belief that contrastive models require large batch sizes and many training epochs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_05490 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Poly-View Contrastive Learning Shidani, Amitis Hjelm, Devon Ramapuram, Jason Webb, Russ Dhekane, Eeshan Gunesh Busbridge, Dan Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Information Theory Contrastive learning typically matches pairs of related views among a number of unrelated negative views. Views can be generated (e.g. by augmentations) or be observed. We investigate matching when there are more than two related views which we call poly-view tasks, and derive new representation learning objectives using information maximization and sufficient statistics. We show that with unlimited computation, one should maximize the number of related views, and with a fixed compute budget, it is beneficial to decrease the number of unique samples whilst increasing the number of views of those samples. In particular, poly-view contrastive models trained for 128 epochs with batch size 256 outperform SimCLR trained for 1024 epochs at batch size 4096 on ImageNet1k, challenging the belief that contrastive models require large batch sizes and many training epochs. |
| title | Poly-View Contrastive Learning |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Information Theory |
| url | https://arxiv.org/abs/2403.05490 |