Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.17607 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917226837180416 |
|---|---|
| author | Okanohara, Daisuke |
| author_facet | Okanohara, Daisuke |
| contents | Learning systems acquire structured internal representations from data, yet classical information-theoretic results state that deterministic transformations do not increase information. This raises a fundamental question: how can learning produce abstraction and insight without violating information-theoretic limits?
We argue that learning is inherently an irreversible process when performed over finite time, and that the realization of epistemic structure necessarily incurs entropy production. To formalize this perspective, we model learning as a transport process in the space of probability distributions over model configurations and introduce an epistemic free-energy framework.
Within this framework, we define the free-energy reduction as a bookkeeping quantity that records the total reduction of epistemic free energy along a learning trajectory. This formulation highlights that realizing such a reduction over finite time necessarily incurs irreversible entropy production.
We then derive the Epistemic Speed Limit (ESL), a finite-time inequality that lower-bounds the minimal entropy production required by any learning process to realize a given distributional transformation. This bound depends only on the Wasserstein distance between initial and final ensemble distributions and is independent of the specific learning algorithm. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_17607 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | A Thermodynamic Theory of Learning I: Irreversible Ensemble Transport and Epistemic Costs Okanohara, Daisuke Machine Learning Learning systems acquire structured internal representations from data, yet classical information-theoretic results state that deterministic transformations do not increase information. This raises a fundamental question: how can learning produce abstraction and insight without violating information-theoretic limits? We argue that learning is inherently an irreversible process when performed over finite time, and that the realization of epistemic structure necessarily incurs entropy production. To formalize this perspective, we model learning as a transport process in the space of probability distributions over model configurations and introduce an epistemic free-energy framework. Within this framework, we define the free-energy reduction as a bookkeeping quantity that records the total reduction of epistemic free energy along a learning trajectory. This formulation highlights that realizing such a reduction over finite time necessarily incurs irreversible entropy production. We then derive the Epistemic Speed Limit (ESL), a finite-time inequality that lower-bounds the minimal entropy production required by any learning process to realize a given distributional transformation. This bound depends only on the Wasserstein distance between initial and final ensemble distributions and is independent of the specific learning algorithm. |
| title | A Thermodynamic Theory of Learning I: Irreversible Ensemble Transport and Epistemic Costs |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2601.17607 |