Neural Thermodynamic Laws for Large Language Model Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909611369431040 |
|---|---|
| author | Liu, Ziming Liu, Yizhou Gore, Jeff Tegmark, Max |
| author_facet | Liu, Ziming Liu, Yizhou Gore, Jeff Tegmark, Max |
| contents | Beyond neural scaling laws, little is known about the laws underlying large language models (LLMs). We introduce Neural Thermodynamic Laws (NTL) -- a new framework that offers fresh insights into LLM training dynamics. On the theoretical side, we demonstrate that key thermodynamic quantities (e.g., temperature, entropy, heat capacity, thermal conduction) and classical thermodynamic principles (e.g., the three laws of thermodynamics and the equipartition theorem) naturally emerge under river-valley loss landscape assumptions. On the practical side, this scientific perspective yields intuitive guidelines for designing learning rate schedules. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_10559 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Neural Thermodynamic Laws for Large Language Model Training Liu, Ziming Liu, Yizhou Gore, Jeff Tegmark, Max Machine Learning Artificial Intelligence Data Analysis, Statistics and Probability Beyond neural scaling laws, little is known about the laws underlying large language models (LLMs). We introduce Neural Thermodynamic Laws (NTL) -- a new framework that offers fresh insights into LLM training dynamics. On the theoretical side, we demonstrate that key thermodynamic quantities (e.g., temperature, entropy, heat capacity, thermal conduction) and classical thermodynamic principles (e.g., the three laws of thermodynamics and the equipartition theorem) naturally emerge under river-valley loss landscape assumptions. On the practical side, this scientific perspective yields intuitive guidelines for designing learning rate schedules. |
| title | Neural Thermodynamic Laws for Large Language Model Training |
| topic | Machine Learning Artificial Intelligence Data Analysis, Statistics and Probability |
| url | https://arxiv.org/abs/2505.10559 |