Probing the Latent Hierarchical Structure of Data via Diffusion Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912252907487232 |
|---|---|
| author | Sclocchi, Antonio Favero, Alessandro Levi, Noam Itzhak Wyart, Matthieu |
| author_facet | Sclocchi, Antonio Favero, Alessandro Levi, Noam Itzhak Wyart, Matthieu |
| contents | High-dimensional data must be highly structured to be learnable. Although the compositional and hierarchical nature of data is often put forward to explain learnability, quantitative measurements establishing these properties are scarce. Likewise, accessing the latent variables underlying such a data structure remains a challenge. In this work, we show that forward-backward experiments in diffusion-based models, where data is noised and then denoised to generate new samples, are a promising tool to probe the latent structure of data. We predict in simple hierarchical models that, in this process, changes in data occur by correlated chunks, with a length scale that diverges at a noise level where a phase transition is known to take place. Remarkably, we confirm this prediction in both text and image datasets using state-of-the-art diffusion models. Our results show how latent variable changes manifest in the data and establish how to measure these effects in real data using diffusion models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_13770 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Probing the Latent Hierarchical Structure of Data via Diffusion Models Sclocchi, Antonio Favero, Alessandro Levi, Noam Itzhak Wyart, Matthieu Machine Learning Disordered Systems and Neural Networks High-dimensional data must be highly structured to be learnable. Although the compositional and hierarchical nature of data is often put forward to explain learnability, quantitative measurements establishing these properties are scarce. Likewise, accessing the latent variables underlying such a data structure remains a challenge. In this work, we show that forward-backward experiments in diffusion-based models, where data is noised and then denoised to generate new samples, are a promising tool to probe the latent structure of data. We predict in simple hierarchical models that, in this process, changes in data occur by correlated chunks, with a length scale that diverges at a noise level where a phase transition is known to take place. Remarkably, we confirm this prediction in both text and image datasets using state-of-the-art diffusion models. Our results show how latent variable changes manifest in the data and establish how to measure these effects in real data using diffusion models. |
| title | Probing the Latent Hierarchical Structure of Data via Diffusion Models |
| topic | Machine Learning Disordered Systems and Neural Networks |
| url | https://arxiv.org/abs/2410.13770 |