What Prompts Don't Say: Understanding and Managing Underspecification in LLM Prompts
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917433858588672 |
|---|---|
| author | Yang, Chenyang Shi, Yike Ma, Qianou Liu, Michael Xieyang Kästner, Christian Wu, Tongshuang |
| author_facet | Yang, Chenyang Shi, Yike Ma, Qianou Liu, Michael Xieyang Kästner, Christian Wu, Tongshuang |
| contents | Prompt underspecification is a common challenge when interacting with LLMs. In this paper, we present an in-depth analysis of this problem, showing that while LLMs can often infer unspecified requirements by default (41.1%), such behavior is fragile: Under-specified prompts are 2x as likely to regress across model or prompt changes, sometimes with accuracy drops exceeding 20%. This instability makes it difficult to reliably build LLM applications. Moreover, simply specifying all requirements does not consistently help, as models have limited instruction-following ability and requirements can conflict. Standard prompt optimizers likewise provide little benefit. To address these issues, we propose requirements-aware prompt optimization mechanisms that improve performance by 4.8% on average over baselines. We further advocate for a systematic process of proactive requirements discovery, evaluation, and monitoring to better manage prompt underspecification in practice. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_13360 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | What Prompts Don't Say: Understanding and Managing Underspecification in LLM Prompts Yang, Chenyang Shi, Yike Ma, Qianou Liu, Michael Xieyang Kästner, Christian Wu, Tongshuang Computation and Language Software Engineering Prompt underspecification is a common challenge when interacting with LLMs. In this paper, we present an in-depth analysis of this problem, showing that while LLMs can often infer unspecified requirements by default (41.1%), such behavior is fragile: Under-specified prompts are 2x as likely to regress across model or prompt changes, sometimes with accuracy drops exceeding 20%. This instability makes it difficult to reliably build LLM applications. Moreover, simply specifying all requirements does not consistently help, as models have limited instruction-following ability and requirements can conflict. Standard prompt optimizers likewise provide little benefit. To address these issues, we propose requirements-aware prompt optimization mechanisms that improve performance by 4.8% on average over baselines. We further advocate for a systematic process of proactive requirements discovery, evaluation, and monitoring to better manage prompt underspecification in practice. |
| title | What Prompts Don't Say: Understanding and Managing Underspecification in LLM Prompts |
| topic | Computation and Language Software Engineering |
| url | https://arxiv.org/abs/2505.13360 |