Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.05536 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913009717215232 |
|---|---|
| author | Yang, Zhongxin Bao, Chun Bin, Yuanwei Yang, Xiang I. A. Chen, Shiyi |
| author_facet | Yang, Zhongxin Bao, Chun Bin, Yuanwei Yang, Xiang I. A. Chen, Shiyi |
| contents | Natural language is a complex system that exhibits robust statistical regularities. Here, we represent text as a trajectory in a high-dimensional embedding space generated by transformer-based language models, and quantify scale-dependent fluctuations along the token sequence using an embedding-step signal. Across multiple languages and corpora, the resulting power spectrum exhibits a robust power law with an exponent close to $5/3$ over an extended frequency range. This scaling is observed consistently in contextual embeddings from both human-written and AI-generated text, but is absent in static word embeddings and is disrupted by randomization of token order. These results show that the observed scaling reflects multiscale, context-dependent organization rather than lexical statistics alone. By analogy with the Kolmogorov spectrum in turbulence, our findings suggest that semantic information is integrated in a scale-free, self-similar manner across linguistic scales, and provide a quantitative, model-agnostic benchmark for studying complex structure in language representations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_05536 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Turbulence-like 5/3 spectral scaling in contextual representations of language as a complex system Yang, Zhongxin Bao, Chun Bin, Yuanwei Yang, Xiang I. A. Chen, Shiyi Computation and Language Artificial Intelligence Natural language is a complex system that exhibits robust statistical regularities. Here, we represent text as a trajectory in a high-dimensional embedding space generated by transformer-based language models, and quantify scale-dependent fluctuations along the token sequence using an embedding-step signal. Across multiple languages and corpora, the resulting power spectrum exhibits a robust power law with an exponent close to $5/3$ over an extended frequency range. This scaling is observed consistently in contextual embeddings from both human-written and AI-generated text, but is absent in static word embeddings and is disrupted by randomization of token order. These results show that the observed scaling reflects multiscale, context-dependent organization rather than lexical statistics alone. By analogy with the Kolmogorov spectrum in turbulence, our findings suggest that semantic information is integrated in a scale-free, self-similar manner across linguistic scales, and provide a quantitative, model-agnostic benchmark for studying complex structure in language representations. |
| title | Turbulence-like 5/3 spectral scaling in contextual representations of language as a complex system |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2604.05536 |