Stories in Space: In-Context Learning Trajectories in Conceptual Belief Space
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918497864384512 |
|---|---|
| author | Bigelow, Eric Sarfati, Raphaël Wurgaft, Daniel Lewis, Owen McGrath, Thomas Merullo, Jack Geiger, Atticus Lubana, Ekdeep Singh |
| author_facet | Bigelow, Eric Sarfati, Raphaël Wurgaft, Daniel Lewis, Owen McGrath, Thomas Merullo, Jack Geiger, Atticus Lubana, Ekdeep Singh |
| contents | Large Language Models (LLMs) update their behavior in context, which can be viewed as a form of Bayesian inference. However, the structure of the latent hypothesis space over which this inference operates remains unclear. In this work, we propose that LLMs assign beliefs over a low-dimensional geometric space - a conceptual belief space - and that in-context learning corresponds to a trajectory through this space as beliefs are updated over time. Using story understanding as a natural setting for dynamic belief updating, we combine behavioral and representational analyses to study these trajectories. We find that (1) belief updates are well-described as trajectories on low-dimensional, structured manifolds; (2) this structure is reflected consistently in both model behavior and internal representations and can be decoded with simple linear probes to predict behavior; and (3) interventions on these representations causally steer belief trajectories, with effects that can be predicted from the geometry of the conceptual space. Together, our results provide a geometric account of belief dynamics in LLMs, grounding Bayesian interpretations of in-context learning in structured conceptual representations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_12412 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Stories in Space: In-Context Learning Trajectories in Conceptual Belief Space Bigelow, Eric Sarfati, Raphaël Wurgaft, Daniel Lewis, Owen McGrath, Thomas Merullo, Jack Geiger, Atticus Lubana, Ekdeep Singh Computation and Language Artificial Intelligence Machine Learning Large Language Models (LLMs) update their behavior in context, which can be viewed as a form of Bayesian inference. However, the structure of the latent hypothesis space over which this inference operates remains unclear. In this work, we propose that LLMs assign beliefs over a low-dimensional geometric space - a conceptual belief space - and that in-context learning corresponds to a trajectory through this space as beliefs are updated over time. Using story understanding as a natural setting for dynamic belief updating, we combine behavioral and representational analyses to study these trajectories. We find that (1) belief updates are well-described as trajectories on low-dimensional, structured manifolds; (2) this structure is reflected consistently in both model behavior and internal representations and can be decoded with simple linear probes to predict behavior; and (3) interventions on these representations causally steer belief trajectories, with effects that can be predicted from the geometry of the conceptual space. Together, our results provide a geometric account of belief dynamics in LLMs, grounding Bayesian interpretations of in-context learning in structured conceptual representations. |
| title | Stories in Space: In-Context Learning Trajectories in Conceptual Belief Space |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2605.12412 |