What Are the Odds? Language Models Are Capable of Probabilistic Reasoning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909330162319360 |
|---|---|
| author | Paruchuri, Akshay Garrison, Jake Liao, Shun Hernandez, John Sunshine, Jacob Althoff, Tim Liu, Xin McDuff, Daniel |
| author_facet | Paruchuri, Akshay Garrison, Jake Liao, Shun Hernandez, John Sunshine, Jacob Althoff, Tim Liu, Xin McDuff, Daniel |
| contents | Language models (LM) are capable of remarkably complex linguistic tasks; however, numerical reasoning is an area in which they frequently struggle. An important but rarely evaluated form of reasoning is understanding probability distributions. In this paper, we focus on evaluating the probabilistic reasoning capabilities of LMs using idealized and real-world statistical distributions. We perform a systematic evaluation of state-of-the-art LMs on three tasks: estimating percentiles, drawing samples, and calculating probabilities. We evaluate three ways to provide context to LMs 1) anchoring examples from within a distribution or family of distributions, 2) real-world context, 3) summary statistics on which to base a Normal approximation. Models can make inferences about distributions, and can be further aided by the incorporation of real-world context, example shots and simplified assumptions, even if these assumptions are incorrect or misspecified. To conduct this work, we developed a comprehensive benchmark distribution dataset with associated question-answer pairs that we have released publicly. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_12830 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | What Are the Odds? Language Models Are Capable of Probabilistic Reasoning Paruchuri, Akshay Garrison, Jake Liao, Shun Hernandez, John Sunshine, Jacob Althoff, Tim Liu, Xin McDuff, Daniel Computation and Language Language models (LM) are capable of remarkably complex linguistic tasks; however, numerical reasoning is an area in which they frequently struggle. An important but rarely evaluated form of reasoning is understanding probability distributions. In this paper, we focus on evaluating the probabilistic reasoning capabilities of LMs using idealized and real-world statistical distributions. We perform a systematic evaluation of state-of-the-art LMs on three tasks: estimating percentiles, drawing samples, and calculating probabilities. We evaluate three ways to provide context to LMs 1) anchoring examples from within a distribution or family of distributions, 2) real-world context, 3) summary statistics on which to base a Normal approximation. Models can make inferences about distributions, and can be further aided by the incorporation of real-world context, example shots and simplified assumptions, even if these assumptions are incorrect or misspecified. To conduct this work, we developed a comprehensive benchmark distribution dataset with associated question-answer pairs that we have released publicly. |
| title | What Are the Odds? Language Models Are Capable of Probabilistic Reasoning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.12830 |