When Softmax Fails at the Top: Extreme Value Corrections for InfoNCE
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917549446266880 |
|---|---|
| author | Erol, Melihcan Evren, Suat Ozel, Oktay Morgan, Alexander Ryu, Jongha Jon Zheng, Lizhong |
| author_facet | Erol, Melihcan Evren, Suat Ozel, Oktay Morgan, Alexander Ryu, Jongha Jon Zheng, Lizhong |
| contents | InfoNCE is the standard contrastive learning objective, but its softmax form is not only a computational convenience: it also encodes a statistical assumption about how the top-scoring example is selected. Using extreme value theory, we show that this assumption is often misaligned with the normalized embedding setting used in modern contrastive learning. Motivated by this mismatch, we propose \textsc{WEINCE}, a simple modification of InfoNCE that uses anchor-wise online batch statistics to blend the usual softmax logits with an endpoint shortfall correction, adding no trainable parameters. Across five vision benchmarks, \textsc{WEINCE} yields consistent improvements in frozen-feature evaluation. These results show that a more faithful statistical treatment of hard negatives can improve contrastive objectives. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_00262 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | When Softmax Fails at the Top: Extreme Value Corrections for InfoNCE Erol, Melihcan Evren, Suat Ozel, Oktay Morgan, Alexander Ryu, Jongha Jon Zheng, Lizhong Machine Learning Artificial Intelligence Applications InfoNCE is the standard contrastive learning objective, but its softmax form is not only a computational convenience: it also encodes a statistical assumption about how the top-scoring example is selected. Using extreme value theory, we show that this assumption is often misaligned with the normalized embedding setting used in modern contrastive learning. Motivated by this mismatch, we propose \textsc{WEINCE}, a simple modification of InfoNCE that uses anchor-wise online batch statistics to blend the usual softmax logits with an endpoint shortfall correction, adding no trainable parameters. Across five vision benchmarks, \textsc{WEINCE} yields consistent improvements in frozen-feature evaluation. These results show that a more faithful statistical treatment of hard negatives can improve contrastive objectives. |
| title | When Softmax Fails at the Top: Extreme Value Corrections for InfoNCE |
| topic | Machine Learning Artificial Intelligence Applications |
| url | https://arxiv.org/abs/2606.00262 |