Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.06808 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912418812133376 |
|---|---|
| author | Michaelov, James A. Estacio, Reeka Zhang, Zhien Bergen, Benjamin K. |
| author_facet | Michaelov, James A. Estacio, Reeka Zhang, Zhien Bergen, Benjamin K. |
| contents | Can language models reliably predict that possible events are more likely than merely improbable ones? By teasing apart possibility, typicality, and contextual relatedness, we show that despite the results of previous work, language models' ability to do this is far from robust. In fact, under certain conditions, all models tested - including Llama 3, Gemma 2, and Mistral NeMo - perform at worse-than-chance level, assigning higher probabilities to impossible sentences such as 'the car was given a parking ticket by the brake' than to merely unlikely sentences such as 'the car was given a parking ticket by the explorer'. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_06808 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Not quite Sherlock Holmes: Language model predictions do not reliably differentiate impossible from improbable events Michaelov, James A. Estacio, Reeka Zhang, Zhien Bergen, Benjamin K. Computation and Language Artificial Intelligence Can language models reliably predict that possible events are more likely than merely improbable ones? By teasing apart possibility, typicality, and contextual relatedness, we show that despite the results of previous work, language models' ability to do this is far from robust. In fact, under certain conditions, all models tested - including Llama 3, Gemma 2, and Mistral NeMo - perform at worse-than-chance level, assigning higher probabilities to impossible sentences such as 'the car was given a parking ticket by the brake' than to merely unlikely sentences such as 'the car was given a parking ticket by the explorer'. |
| title | Not quite Sherlock Holmes: Language model predictions do not reliably differentiate impossible from improbable events |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2506.06808 |