"Noisier" Noise Contrastive Eestimation is (Almost) Maximum Likelihood
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918466785640448 |
|---|---|
| author | Yu, Peiyu Zhang, Dinghuai He, Hengzhi Ma, Xiaojian Xie, Sirui Miao, Ruiyao Lu, Yifan Zhang, Yasi Kong, Deqian Gao, Ruiqi Xie, Jianwen Cheng, Guang Wu, Ying Nian |
| author_facet | Yu, Peiyu Zhang, Dinghuai He, Hengzhi Ma, Xiaojian Xie, Sirui Miao, Ruiyao Lu, Yifan Zhang, Yasi Kong, Deqian Gao, Ruiqi Xie, Jianwen Cheng, Guang Wu, Ying Nian |
| contents | Noise Contrastive Estimation (NCE) has fueled major breakthroughs in representation learning and generative modeling. Yet a long-standing challenge remains: accurately estimating ratios between distributions that differ substantially, which significantly limits the applicability of NCE on modern high-dimensional and multimodal datasets. We revisit this problem from a less explored perspective: the magnitude of the noise distribution. Specifically, we show that with a virtually scaled (\ie, artificially increased) noise magnitude, the gradient of the NCE objective can closely align with that of Maximum Likelihood, enabling a trajectory-wise approximation from NCE to MLE, and faster convergence both theoretically and empirically. Building on this insight, we introduce ``Noisier'' NCE, a simple drop-in modification to vanilla NCE that incurs little to no extra computational cost, while effectively handling density-ratio estimation in challenging regimes where traditional MLE and NCE struggle. Beyond improving classical density-ratio learning, ``Noisier'' NCE proves broadly applicable: it achieves strong results across image modeling, anomaly detection, and offline black-box optimization. On CIFAR-10 and ImageNet64x64 datasets, it yields 10-step and even 1-step samplers that match or surpass state-of-the-art methods, while cutting training iterations by up to half. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_16730 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | "Noisier" Noise Contrastive Eestimation is (Almost) Maximum Likelihood Yu, Peiyu Zhang, Dinghuai He, Hengzhi Ma, Xiaojian Xie, Sirui Miao, Ruiyao Lu, Yifan Zhang, Yasi Kong, Deqian Gao, Ruiqi Xie, Jianwen Cheng, Guang Wu, Ying Nian Machine Learning Artificial Intelligence Applications Noise Contrastive Estimation (NCE) has fueled major breakthroughs in representation learning and generative modeling. Yet a long-standing challenge remains: accurately estimating ratios between distributions that differ substantially, which significantly limits the applicability of NCE on modern high-dimensional and multimodal datasets. We revisit this problem from a less explored perspective: the magnitude of the noise distribution. Specifically, we show that with a virtually scaled (\ie, artificially increased) noise magnitude, the gradient of the NCE objective can closely align with that of Maximum Likelihood, enabling a trajectory-wise approximation from NCE to MLE, and faster convergence both theoretically and empirically. Building on this insight, we introduce ``Noisier'' NCE, a simple drop-in modification to vanilla NCE that incurs little to no extra computational cost, while effectively handling density-ratio estimation in challenging regimes where traditional MLE and NCE struggle. Beyond improving classical density-ratio learning, ``Noisier'' NCE proves broadly applicable: it achieves strong results across image modeling, anomaly detection, and offline black-box optimization. On CIFAR-10 and ImageNet64x64 datasets, it yields 10-step and even 1-step samplers that match or surpass state-of-the-art methods, while cutting training iterations by up to half. |
| title | "Noisier" Noise Contrastive Eestimation is (Almost) Maximum Likelihood |
| topic | Machine Learning Artificial Intelligence Applications |
| url | https://arxiv.org/abs/2405.16730 |