Failure to Mix: Large language models struggle to answer according to desired probability distributions
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918208278102016 |
|---|---|
| author | Yang, Ivy Yuqian Zhang, David Yu |
| author_facet | Yang, Ivy Yuqian Zhang, David Yu |
| contents | Scientific idea generation and selection requires exploration following a target probability distribution. In contrast, current AI benchmarks have objectively correct answers, and training large language models (LLMs) via reinforcement learning against these benchmarks discourages probabilistic exploration. Here, we conducted systematic experiments requesting LLMs to produce outputs following simple probabilistic distributions, and found that all modern LLMs tested grossly fail to follow the distributions. For example, requesting a binary output of "1" 49% of the time produces an answer of "0" nearly 100% of the time. This step function-like behavior of near-exclusively generating the output with marginally highest probability even overrules even strong in-built LLM biases. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_14630 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Failure to Mix: Large language models struggle to answer according to desired probability distributions Yang, Ivy Yuqian Zhang, David Yu Machine Learning Artificial Intelligence Scientific idea generation and selection requires exploration following a target probability distribution. In contrast, current AI benchmarks have objectively correct answers, and training large language models (LLMs) via reinforcement learning against these benchmarks discourages probabilistic exploration. Here, we conducted systematic experiments requesting LLMs to produce outputs following simple probabilistic distributions, and found that all modern LLMs tested grossly fail to follow the distributions. For example, requesting a binary output of "1" 49% of the time produces an answer of "0" nearly 100% of the time. This step function-like behavior of near-exclusively generating the output with marginally highest probability even overrules even strong in-built LLM biases. |
| title | Failure to Mix: Large language models struggle to answer according to desired probability distributions |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2511.14630 |