Losing dimensions: Geometric memorization in generative diffusion
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908878478770176 |
|---|---|
| author | Achilli, Beatrice Ventura, Enrico Silvestri, Gianluigi Pham, Bao Raya, Gabriel Krotov, Dmitry Lucibello, Carlo Ambrogioni, Luca |
| author_facet | Achilli, Beatrice Ventura, Enrico Silvestri, Gianluigi Pham, Bao Raya, Gabriel Krotov, Dmitry Lucibello, Carlo Ambrogioni, Luca |
| contents | Diffusion models power leading generative AI, but when and how they memorize training data, especially on low-dimensional manifolds, remains unclear. We find memorization emerges gradually, not abruptly: as data become scarce, diffusion models experience a smooth collapse where their capacity to vary across independent directions diminishes. Measuring latent dimensionality via the learned score field, we reveal how generative behavior increasingly centers on a few examples while other variations "freeze out". We propose a geometric memorization theory, showing that salient features collapse first, then finer details, leading to near point-wise replication. This mirrors physical systems condensing into a few low-energy configurations. Our theoretical predictions align with both synthetic and real data, identifying geometric memorization as a distinct phase between generalization and exact copying. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_08727 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Losing dimensions: Geometric memorization in generative diffusion Achilli, Beatrice Ventura, Enrico Silvestri, Gianluigi Pham, Bao Raya, Gabriel Krotov, Dmitry Lucibello, Carlo Ambrogioni, Luca Machine Learning Diffusion models power leading generative AI, but when and how they memorize training data, especially on low-dimensional manifolds, remains unclear. We find memorization emerges gradually, not abruptly: as data become scarce, diffusion models experience a smooth collapse where their capacity to vary across independent directions diminishes. Measuring latent dimensionality via the learned score field, we reveal how generative behavior increasingly centers on a few examples while other variations "freeze out". We propose a geometric memorization theory, showing that salient features collapse first, then finer details, leading to near point-wise replication. This mirrors physical systems condensing into a few low-energy configurations. Our theoretical predictions align with both synthetic and real data, identifying geometric memorization as a distinct phase between generalization and exact copying. |
| title | Losing dimensions: Geometric memorization in generative diffusion |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2410.08727 |