Learning is Forgetting: LLM Training As Lossy Compression
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918435770859520 |
|---|---|
| author | Conklin, Henry C. Hosking, Tom Yi-Chern, Tan Gold, Julian Cohen, Jonathan D. Griffiths, Thomas L. Bartolo, Max Goldfarb-Tarrant, Seraphina |
| author_facet | Conklin, Henry C. Hosking, Tom Yi-Chern, Tan Gold, Julian Cohen, Jonathan D. Griffiths, Thomas L. Bartolo, Max Goldfarb-Tarrant, Seraphina |
| contents | Despite the increasing prevalence of large language models (LLMs), we still have a limited understanding of how their representational spaces are structured. This limits our ability to interpret how and what they learn or relate them to learning in humans. We argue LLMs are best seen as an instance of lossy compression, where over training they learn by retaining only information in their training data relevant to their objective(s). We show pre-training results in models that are optimally compressed for next-sequence prediction, approaching the Information Bottleneck bound on compression. Across an array of open weights models, each compresses differently, likely due to differences in the data and training recipes used. However even across different families of LLMs the optimality of a model's compression, and the information present in it, can predict downstream performance on across a wide array of benchmarks, letting us directly link representational structure to actionable insights about model performance. In the general case the work presented here offers a unified Information-Theoretic framing for how these models learn that is deployable at scale. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_07569 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Learning is Forgetting: LLM Training As Lossy Compression Conklin, Henry C. Hosking, Tom Yi-Chern, Tan Gold, Julian Cohen, Jonathan D. Griffiths, Thomas L. Bartolo, Max Goldfarb-Tarrant, Seraphina Machine Learning Artificial Intelligence Computation and Language Information Theory Despite the increasing prevalence of large language models (LLMs), we still have a limited understanding of how their representational spaces are structured. This limits our ability to interpret how and what they learn or relate them to learning in humans. We argue LLMs are best seen as an instance of lossy compression, where over training they learn by retaining only information in their training data relevant to their objective(s). We show pre-training results in models that are optimally compressed for next-sequence prediction, approaching the Information Bottleneck bound on compression. Across an array of open weights models, each compresses differently, likely due to differences in the data and training recipes used. However even across different families of LLMs the optimality of a model's compression, and the information present in it, can predict downstream performance on across a wide array of benchmarks, letting us directly link representational structure to actionable insights about model performance. In the general case the work presented here offers a unified Information-Theoretic framing for how these models learn that is deployable at scale. |
| title | Learning is Forgetting: LLM Training As Lossy Compression |
| topic | Machine Learning Artificial Intelligence Computation and Language Information Theory |
| url | https://arxiv.org/abs/2604.07569 |