Markov Chain Estimation with In-Context Learning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912522076946432 |
|---|---|
| author | Lepage, Simon Mary, Jeremie Picard, David |
| author_facet | Lepage, Simon Mary, Jeremie Picard, David |
| contents | We investigate the capacity of transformers to learn algorithms involving their context while solely being trained using next token prediction. We set up Markov chains with random transition matrices and we train transformers to predict the next token. Matrices used during training and test are different and we show that there is a threshold in transformer size and in training set size above which the model is able to learn to estimate the transition probabilities from its context instead of memorizing the training patterns. Additionally, we show that more involved encoding of the states enables more robust prediction for Markov chains with structures different than those seen during training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_03934 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Markov Chain Estimation with In-Context Learning Lepage, Simon Mary, Jeremie Picard, David Machine Learning 68T07 We investigate the capacity of transformers to learn algorithms involving their context while solely being trained using next token prediction. We set up Markov chains with random transition matrices and we train transformers to predict the next token. Matrices used during training and test are different and we show that there is a threshold in transformer size and in training set size above which the model is able to learn to estimate the transition probabilities from its context instead of memorizing the training patterns. Additionally, we show that more involved encoding of the states enables more robust prediction for Markov chains with structures different than those seen during training. |
| title | Markov Chain Estimation with In-Context Learning |
| topic | Machine Learning 68T07 |
| url | https://arxiv.org/abs/2508.03934 |