Variance reduction of diffusion model's gradients with Taylor approximation-based control variate
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916365994033152 |
|---|---|
| author | Jeha, Paul Grathwohl, Will Andersen, Michael Riis Ek, Carl Henrik Frellsen, Jes |
| author_facet | Jeha, Paul Grathwohl, Will Andersen, Michael Riis Ek, Carl Henrik Frellsen, Jes |
| contents | Score-based models, trained with denoising score matching, are remarkably effective in generating high dimensional data. However, the high variance of their training objective hinders optimisation. We attempt to reduce it with a control variate, derived via a $k$-th order Taylor expansion on the training objective and its gradient. We prove an equivalence between the two and demonstrate empirically the effectiveness of our approach on a low dimensional problem setting; and study its effect on larger problems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_12270 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Variance reduction of diffusion model's gradients with Taylor approximation-based control variate Jeha, Paul Grathwohl, Will Andersen, Michael Riis Ek, Carl Henrik Frellsen, Jes Machine Learning Artificial Intelligence Score-based models, trained with denoising score matching, are remarkably effective in generating high dimensional data. However, the high variance of their training objective hinders optimisation. We attempt to reduce it with a control variate, derived via a $k$-th order Taylor expansion on the training objective and its gradient. We prove an equivalence between the two and demonstrate empirically the effectiveness of our approach on a low dimensional problem setting; and study its effect on larger problems. |
| title | Variance reduction of diffusion model's gradients with Taylor approximation-based control variate |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2408.12270 |