Why Heuristic Weighting Works: A Theoretical Analysis of Denoising Score Matching
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911088531996672 |
|---|---|
| author | Zhang, Juyan Newbury, Rhys Zhang, Xinyang Tran, Tin Kulic, Dana Burke, Michael |
| author_facet | Zhang, Juyan Newbury, Rhys Zhang, Xinyang Tran, Tin Kulic, Dana Burke, Michael |
| contents | Score matching enables the estimation of the gradient of a data distribution, a key component in denoising diffusion models used to recover clean data from corrupted inputs. In prior work, a heuristic weighting function has been used for the denoising score matching loss without formal justification. In this work, we demonstrate that heteroskedasticity is an inherent property of the denoising score matching objective. This insight leads to a principled derivation of optimal weighting functions for generalized, arbitrary-order denoising score matching losses, without requiring assumptions about the noise distribution. Among these, the first-order formulation is especially relevant to diffusion models. We show that the widely used heuristical weighting function arises as a first-order Taylor approximation to the trace of the expected optimal weighting. We further provide theoretical and empirical comparisons, revealing that the heuristical weighting, despite its simplicity, can achieve lower variance than the optimal weighting with respect to parameter gradients, which can facilitate more stable and efficient training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_01597 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Why Heuristic Weighting Works: A Theoretical Analysis of Denoising Score Matching Zhang, Juyan Newbury, Rhys Zhang, Xinyang Tran, Tin Kulic, Dana Burke, Michael Machine Learning Applications Score matching enables the estimation of the gradient of a data distribution, a key component in denoising diffusion models used to recover clean data from corrupted inputs. In prior work, a heuristic weighting function has been used for the denoising score matching loss without formal justification. In this work, we demonstrate that heteroskedasticity is an inherent property of the denoising score matching objective. This insight leads to a principled derivation of optimal weighting functions for generalized, arbitrary-order denoising score matching losses, without requiring assumptions about the noise distribution. Among these, the first-order formulation is especially relevant to diffusion models. We show that the widely used heuristical weighting function arises as a first-order Taylor approximation to the trace of the expected optimal weighting. We further provide theoretical and empirical comparisons, revealing that the heuristical weighting, despite its simplicity, can achieve lower variance than the optimal weighting with respect to parameter gradients, which can facilitate more stable and efficient training. |
| title | Why Heuristic Weighting Works: A Theoretical Analysis of Denoising Score Matching |
| topic | Machine Learning Applications |
| url | https://arxiv.org/abs/2508.01597 |