Lines of Thought in Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910826853564416 |
|---|---|
| author | Sarfati, Raphaël Liu, Toni J. B. Boullé, Nicolas Earls, Christopher J. |
| author_facet | Sarfati, Raphaël Liu, Toni J. B. Boullé, Nicolas Earls, Christopher J. |
| contents | Large Language Models achieve next-token prediction by transporting a vectorized piece of text (prompt) across an accompanying embedding space under the action of successive transformer layers. The resulting high-dimensional trajectories realize different contextualization, or 'thinking', steps, and fully determine the output probability distribution. We aim to characterize the statistical properties of ensembles of these 'lines of thought.' We observe that independent trajectories cluster along a low-dimensional, non-Euclidean manifold, and that their path can be well approximated by a stochastic equation with few parameters extracted from data. We find it remarkable that the vast complexity of such large models can be reduced to a much simpler form, and we reflect on implications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_01545 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Lines of Thought in Large Language Models Sarfati, Raphaël Liu, Toni J. B. Boullé, Nicolas Earls, Christopher J. Machine Learning Data Analysis, Statistics and Probability Large Language Models achieve next-token prediction by transporting a vectorized piece of text (prompt) across an accompanying embedding space under the action of successive transformer layers. The resulting high-dimensional trajectories realize different contextualization, or 'thinking', steps, and fully determine the output probability distribution. We aim to characterize the statistical properties of ensembles of these 'lines of thought.' We observe that independent trajectories cluster along a low-dimensional, non-Euclidean manifold, and that their path can be well approximated by a stochastic equation with few parameters extracted from data. We find it remarkable that the vast complexity of such large models can be reduced to a much simpler form, and we reflect on implications. |
| title | Lines of Thought in Large Language Models |
| topic | Machine Learning Data Analysis, Statistics and Probability |
| url | https://arxiv.org/abs/2410.01545 |