Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.21339 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918312959541248 |
|---|---|
| author | Haase, Jennifer Gonnermann-Müller, Jana Hanel, Paul H. P. Leins, Nicolas Kosch, Thomas Mendling, Jan Pokutta, Sebastian |
| author_facet | Haase, Jennifer Gonnermann-Müller, Jana Hanel, Paul H. P. Leins, Nicolas Kosch, Thomas Mendling, Jan Pokutta, Sebastian |
| contents | How much of LLM output variance is explained by prompts versus model choice versus stochasticity through sampling? We answer this by evaluating 12 LLMs on 10 creativity prompts with 100 samples each (N = 12,000). For output quality (originality), prompts explain 36.43% of variance, comparable to model choice (40.94%). But for output quantity (fluency), model choice (51.25%) and within-LLM variance (33.70%) dominate, with prompts explaining only 4.22%. Prompts are powerful levers for steering output quality, but given the substantial within-LLM variance (10-34%), single-sample evaluations risk conflating sampling noise with genuine prompt or model effects. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_21339 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks Haase, Jennifer Gonnermann-Müller, Jana Hanel, Paul H. P. Leins, Nicolas Kosch, Thomas Mendling, Jan Pokutta, Sebastian Artificial Intelligence How much of LLM output variance is explained by prompts versus model choice versus stochasticity through sampling? We answer this by evaluating 12 LLMs on 10 creativity prompts with 100 samples each (N = 12,000). For output quality (originality), prompts explain 36.43% of variance, comparable to model choice (40.94%). But for output quantity (fluency), model choice (51.25%) and within-LLM variance (33.70%) dominate, with prompts explaining only 4.22%. Prompts are powerful levers for steering output quality, but given the substantial within-LLM variance (10-34%), single-sample evaluations risk conflating sampling noise with genuine prompt or model effects. |
| title | Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2601.21339 |