Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.07233 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917228944818176 |
|---|---|
| author | Behera, Agnish Kumar Lamtyugina, Alexandra Nandy, Aditya Goto, Daiki Floyd, Carlos Vaikuntanathan, Suriyanarayanan |
| author_facet | Behera, Agnish Kumar Lamtyugina, Alexandra Nandy, Aditya Goto, Daiki Floyd, Carlos Vaikuntanathan, Suriyanarayanan |
| contents | Generative diffusion models have emerged as powerful tools for sampling high-dimensional distributions, yet they typically rely on white gaussian noise and noise schedules to destroy and reconstruct information. Here, we demonstrate that driving the generative process out of equilibrium using active, temporally correlated noise sources fundamentally alters the information thermodynamics of the system. We show that coupling the data to an active non-Markovian bath creates a `memory effect' where high-level semantic information (such as class identity or molecular metastability) is stored in the temporal correlations of auxiliary degrees of freedom. Using Fisher information analysis, we prove that this active mechanism significantly retards the rate of information decay compared to passive Brownian motion. Crucially, this memory effect facilitates an earlier and more robust symmetry breaking (speciation) during the reverse generative process, allowing the system to resolve multi-scale structures, reminiscent of metastable states in molecular configurations that are washed out in the typical noising processes. Our results suggest that non-equilibrium protocols, inspired by active matter physics, offer a thermodynamically distinct and potentially advantageous pathway for recovering high-dimensional energy landscapes using generative diffusion. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_07233 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Non-equilibrium active noise enhances generative memory in diffusion models Behera, Agnish Kumar Lamtyugina, Alexandra Nandy, Aditya Goto, Daiki Floyd, Carlos Vaikuntanathan, Suriyanarayanan Machine Learning Disordered Systems and Neural Networks Generative diffusion models have emerged as powerful tools for sampling high-dimensional distributions, yet they typically rely on white gaussian noise and noise schedules to destroy and reconstruct information. Here, we demonstrate that driving the generative process out of equilibrium using active, temporally correlated noise sources fundamentally alters the information thermodynamics of the system. We show that coupling the data to an active non-Markovian bath creates a `memory effect' where high-level semantic information (such as class identity or molecular metastability) is stored in the temporal correlations of auxiliary degrees of freedom. Using Fisher information analysis, we prove that this active mechanism significantly retards the rate of information decay compared to passive Brownian motion. Crucially, this memory effect facilitates an earlier and more robust symmetry breaking (speciation) during the reverse generative process, allowing the system to resolve multi-scale structures, reminiscent of metastable states in molecular configurations that are washed out in the typical noising processes. Our results suggest that non-equilibrium protocols, inspired by active matter physics, offer a thermodynamically distinct and potentially advantageous pathway for recovering high-dimensional energy landscapes using generative diffusion. |
| title | Non-equilibrium active noise enhances generative memory in diffusion models |
| topic | Machine Learning Disordered Systems and Neural Networks |
| url | https://arxiv.org/abs/2411.07233 |