Momentum Contrastive Learning with Enhanced Negative Sampling and Hard Negative Filtering
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917904044261376 |
|---|---|
| author | Hoang, Duy Ngo, Huy Pham, Khoi Nguyen, Tri Bao, Gia Phan, Huy |
| author_facet | Hoang, Duy Ngo, Huy Pham, Khoi Nguyen, Tri Bao, Gia Phan, Huy |
| contents | Contrastive learning has become pivotal in unsupervised representation learning, with frameworks like Momentum Contrast (MoCo) effectively utilizing large negative sample sets to extract discriminative features. However, traditional approaches often overlook the full potential of key embeddings and are susceptible to performance degradation from noisy negative samples in the memory bank. This study addresses these challenges by proposing an enhanced contrastive learning framework that incorporates two key innovations. First, we introduce a dual-view loss function, which ensures balanced optimization of both query and key embeddings, improving representation quality. Second, we develop a selective negative sampling strategy that emphasizes the most challenging negatives based on cosine similarity, mitigating the impact of noise and enhancing feature discrimination. Extensive experiments demonstrate that our framework achieves superior performance on downstream tasks, delivering robust and well-structured representations. These results highlight the potential of optimized contrastive mechanisms to advance unsupervised learning and extend its applicability across domains such as computer vision and natural language processing |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_16360 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Momentum Contrastive Learning with Enhanced Negative Sampling and Hard Negative Filtering Hoang, Duy Ngo, Huy Pham, Khoi Nguyen, Tri Bao, Gia Phan, Huy Machine Learning Artificial Intelligence Contrastive learning has become pivotal in unsupervised representation learning, with frameworks like Momentum Contrast (MoCo) effectively utilizing large negative sample sets to extract discriminative features. However, traditional approaches often overlook the full potential of key embeddings and are susceptible to performance degradation from noisy negative samples in the memory bank. This study addresses these challenges by proposing an enhanced contrastive learning framework that incorporates two key innovations. First, we introduce a dual-view loss function, which ensures balanced optimization of both query and key embeddings, improving representation quality. Second, we develop a selective negative sampling strategy that emphasizes the most challenging negatives based on cosine similarity, mitigating the impact of noise and enhancing feature discrimination. Extensive experiments demonstrate that our framework achieves superior performance on downstream tasks, delivering robust and well-structured representations. These results highlight the potential of optimized contrastive mechanisms to advance unsupervised learning and extend its applicability across domains such as computer vision and natural language processing |
| title | Momentum Contrastive Learning with Enhanced Negative Sampling and Hard Negative Filtering |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2501.16360 |