Are LLM Belief Updates Consistent with Bayes' Theorem?
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912499199115264 |
|---|---|
| author | Imran, Sohaib Kendiukhov, Ihor Broerman, Matthew Thomas, Aditya Campanella, Riccardo Lamb, Rob Atkinson, Peter M. |
| author_facet | Imran, Sohaib Kendiukhov, Ihor Broerman, Matthew Thomas, Aditya Campanella, Riccardo Lamb, Rob Atkinson, Peter M. |
| contents | Do larger and more capable language models learn to update their "beliefs" about propositions more consistently with Bayes' theorem when presented with evidence in-context? To test this, we formulate a Bayesian Coherence Coefficient (BCC) metric and generate a dataset with which to measure the BCC. We measure BCC for multiple pre-trained-only language models across five model families, comparing against the number of model parameters, the amount of training data, and model scores on common benchmarks. Our results provide evidence for our hypothesis that larger and more capable pre-trained language models assign credences that are more coherent with Bayes' theorem. These results have important implications for our understanding and governance of LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_17951 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Are LLM Belief Updates Consistent with Bayes' Theorem? Imran, Sohaib Kendiukhov, Ihor Broerman, Matthew Thomas, Aditya Campanella, Riccardo Lamb, Rob Atkinson, Peter M. Computation and Language Artificial Intelligence Do larger and more capable language models learn to update their "beliefs" about propositions more consistently with Bayes' theorem when presented with evidence in-context? To test this, we formulate a Bayesian Coherence Coefficient (BCC) metric and generate a dataset with which to measure the BCC. We measure BCC for multiple pre-trained-only language models across five model families, comparing against the number of model parameters, the amount of training data, and model scores on common benchmarks. Our results provide evidence for our hypothesis that larger and more capable pre-trained language models assign credences that are more coherent with Bayes' theorem. These results have important implications for our understanding and governance of LLMs. |
| title | Are LLM Belief Updates Consistent with Bayes' Theorem? |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2507.17951 |