Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.11275 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909349944754176 |
|---|---|
| author | Boffi, Nicholas M. Jacot, Arthur Tu, Stephen Ziemann, Ingvar |
| author_facet | Boffi, Nicholas M. Jacot, Arthur Tu, Stephen Ziemann, Ingvar |
| contents | Diffusion-based generative models provide a powerful framework for learning to sample from a complex target distribution. The remarkable empirical success of these models applied to high-dimensional signals, including images and video, stands in stark contrast to classical results highlighting the curse of dimensionality for distribution recovery. In this work, we take a step towards understanding this gap through a careful analysis of learning diffusion models over the Barron space of single layer neural networks. In particular, we show that these shallow models provably adapt to simple forms of low dimensional structure, thereby avoiding the curse of dimensionality. We combine our results with recent analyses of sampling with diffusion models to provide an end-to-end sample complexity bound for learning to sample from structured distributions. Importantly, our results do not require specialized architectures tailored to particular latent structures, and instead rely on the low-index structure of the Barron space to adapt to the underlying distribution. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_11275 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Shallow diffusion networks provably learn hidden low-dimensional structure Boffi, Nicholas M. Jacot, Arthur Tu, Stephen Ziemann, Ingvar Machine Learning Diffusion-based generative models provide a powerful framework for learning to sample from a complex target distribution. The remarkable empirical success of these models applied to high-dimensional signals, including images and video, stands in stark contrast to classical results highlighting the curse of dimensionality for distribution recovery. In this work, we take a step towards understanding this gap through a careful analysis of learning diffusion models over the Barron space of single layer neural networks. In particular, we show that these shallow models provably adapt to simple forms of low dimensional structure, thereby avoiding the curse of dimensionality. We combine our results with recent analyses of sampling with diffusion models to provide an end-to-end sample complexity bound for learning to sample from structured distributions. Importantly, our results do not require specialized architectures tailored to particular latent structures, and instead rely on the low-index structure of the Barron space to adapt to the underlying distribution. |
| title | Shallow diffusion networks provably learn hidden low-dimensional structure |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2410.11275 |