Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2502.01481 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915826265751552 |
|---|---|
| author | Shi, Jingzhe Ma, Qinwei Liu, Hongyi Zhao, Hang Hwang, Jeng-Neng Li, Lei |
| author_facet | Shi, Jingzhe Ma, Qinwei Liu, Hongyi Zhao, Hang Hwang, Jeng-Neng Li, Lei |
| contents | Long Context Language Models have drawn great attention in the past few years. There has been work discussing the impact of long context on Language Model performance: some find that long irrelevant context could harm performance, while some experimentally summarize loss reduction by relevant long context as Scaling Laws. This calls for a more thorough understanding of how long context impacts Language Modeling. In this work, we (1) propose to use `Intrinsic Entropy' for explaining the impact of context length on language modeling; and (2) conduct experiments on natural language and synthetic data, validating our proposed theoretical assumptions and deductions. Our theoretical framework can provide practical insights such as establishing that training dataset size dictates an optimal context length and bounds context length scaling for certain cases. We hope our work may inspire new long context Language Models, as well as future work studying the physics of Language Models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_01481 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Intrinsic Entropy of Context Length Scaling in LLMs Shi, Jingzhe Ma, Qinwei Liu, Hongyi Zhao, Hang Hwang, Jeng-Neng Li, Lei Machine Learning Computation and Language Long Context Language Models have drawn great attention in the past few years. There has been work discussing the impact of long context on Language Model performance: some find that long irrelevant context could harm performance, while some experimentally summarize loss reduction by relevant long context as Scaling Laws. This calls for a more thorough understanding of how long context impacts Language Modeling. In this work, we (1) propose to use `Intrinsic Entropy' for explaining the impact of context length on language modeling; and (2) conduct experiments on natural language and synthetic data, validating our proposed theoretical assumptions and deductions. Our theoretical framework can provide practical insights such as establishing that training dataset size dictates an optimal context length and bounds context length scaling for certain cases. We hope our work may inspire new long context Language Models, as well as future work studying the physics of Language Models. |
| title | Intrinsic Entropy of Context Length Scaling in LLMs |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2502.01481 |