Lossy Loops: Shannon's DPI and Information Decay in Generative Model Training
Fuente:
Zenodo
Salvato in:
| Autore principale: | |
|---|---|
| Natura: | Recurso digital |
| Lingua: | inglese |
| Pubblicazione: |
Zenodo
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866902000702062592 |
|---|---|
| author | Straňák, Pavel |
| author_facet | Straňák, Pavel |
| contents | <p>Model collapse, the progressive degradation of generative AI performance when trained on synthetic data, poses a critical challenge for modern AI systems. This paper establishes a theoretical framework based on Shannon's Data Processing Inequality (DPI) to explain this phenomenon. We conceptualize generative AI models as lossy communication channels, predicting progressive mutual information decay during iterative training. We derive testable hypotheses for exponential decay rates (λ ∈ [0.2, 0.4] per iteration) and propose mitigation paradigms requiring future validation.</p> <p>See also:<a href="https://doi.org/10.5281/zenodo.15199262"> https://doi.org/10.5281/zenodo.15199262</a> for a related philosophical exploration.</p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_16481493 |
| institution | Zenodo |
| language | eng |
| publishDate | 2025 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | Lossy Loops: Shannon's DPI and Information Decay in Generative Model Training Straňák, Pavel Model collapse, Information theory, Data Processing Inequality, Generative AI, Synthetic data, Mutual information, AI limitations <p>Model collapse, the progressive degradation of generative AI performance when trained on synthetic data, poses a critical challenge for modern AI systems. This paper establishes a theoretical framework based on Shannon's Data Processing Inequality (DPI) to explain this phenomenon. We conceptualize generative AI models as lossy communication channels, predicting progressive mutual information decay during iterative training. We derive testable hypotheses for exponential decay rates (λ ∈ [0.2, 0.4] per iteration) and propose mitigation paradigms requiring future validation.</p> <p>See also:<a href="https://doi.org/10.5281/zenodo.15199262"> https://doi.org/10.5281/zenodo.15199262</a> for a related philosophical exploration.</p> |
| title | Lossy Loops: Shannon's DPI and Information Decay in Generative Model Training |
| topic | Model collapse, Information theory, Data Processing Inequality, Generative AI, Synthetic data, Mutual information, AI limitations |
| url | https://doi.org/10.5281/zenodo.16481493 |