Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918312959541248 |
|---|---|
| author | Haase, Jennifer Gonnermann-Müller, Jana Hanel, Paul H. P. Leins, Nicolas Kosch, Thomas Mendling, Jan Pokutta, Sebastian |
| author_facet | Haase, Jennifer Gonnermann-Müller, Jana Hanel, Paul H. P. Leins, Nicolas Kosch, Thomas Mendling, Jan Pokutta, Sebastian |
| contents | How much of LLM output variance is explained by prompts versus model choice versus stochasticity through sampling? We answer this by evaluating 12 LLMs on 10 creativity prompts with 100 samples each (N = 12,000). For output quality (originality), prompts explain 36.43% of variance, comparable to model choice (40.94%). But for output quantity (fluency), model choice (51.25%) and within-LLM variance (33.70%) dominate, with prompts explaining only 4.22%. Prompts are powerful levers for steering output quality, but given the substantial within-LLM variance (10-34%), single-sample evaluations risk conflating sampling noise with genuine prompt or model effects. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_21339 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks Haase, Jennifer Gonnermann-Müller, Jana Hanel, Paul H. P. Leins, Nicolas Kosch, Thomas Mendling, Jan Pokutta, Sebastian Artificial Intelligence How much of LLM output variance is explained by prompts versus model choice versus stochasticity through sampling? We answer this by evaluating 12 LLMs on 10 creativity prompts with 100 samples each (N = 12,000). For output quality (originality), prompts explain 36.43% of variance, comparable to model choice (40.94%). But for output quantity (fluency), model choice (51.25%) and within-LLM variance (33.70%) dominate, with prompts explaining only 4.22%. Prompts are powerful levers for steering output quality, but given the substantial within-LLM variance (10-34%), single-sample evaluations risk conflating sampling noise with genuine prompt or model effects. |
| title | Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2601.21339 |