What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909595111260160 |
|---|---|
| author | Ma, Qianou Peng, Weirui Yang, Chenyang Shen, Hua Koedinger, Kenneth Wu, Tongshuang |
| author_facet | Ma, Qianou Peng, Weirui Yang, Chenyang Shen, Hua Koedinger, Kenneth Wu, Tongshuang |
| contents | Prompting LLMs for complex tasks (e.g., building a trip advisor chatbot) needs humans to clearly articulate customized requirements (e.g., "start the response with a tl;dr"). However, existing prompt engineering instructions often lack focused training on requirement articulation and instead tend to emphasize increasingly automatable strategies (e.g., tricks like adding role-plays and "think step-by-step"). To address the gap, we introduce Requirement-Oriented Prompt Engineering (ROPE), a paradigm that focuses human attention on generating clear, complete requirements during prompting. We implement ROPE through an assessment and training suite that provides deliberate practice with LLM-generated feedback. In a randomized controlled experiment with 30 novices, ROPE significantly outperforms conventional prompt engineering training (20% vs. 1% gains), a gap that automatic prompt optimization cannot close. Furthermore, we demonstrate a direct correlation between the quality of input requirements and LLM outputs. Our work paves the way to empower more end-users to build complex LLM applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_08775 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use Ma, Qianou Peng, Weirui Yang, Chenyang Shen, Hua Koedinger, Kenneth Wu, Tongshuang Human-Computer Interaction Artificial Intelligence Prompting LLMs for complex tasks (e.g., building a trip advisor chatbot) needs humans to clearly articulate customized requirements (e.g., "start the response with a tl;dr"). However, existing prompt engineering instructions often lack focused training on requirement articulation and instead tend to emphasize increasingly automatable strategies (e.g., tricks like adding role-plays and "think step-by-step"). To address the gap, we introduce Requirement-Oriented Prompt Engineering (ROPE), a paradigm that focuses human attention on generating clear, complete requirements during prompting. We implement ROPE through an assessment and training suite that provides deliberate practice with LLM-generated feedback. In a randomized controlled experiment with 30 novices, ROPE significantly outperforms conventional prompt engineering training (20% vs. 1% gains), a gap that automatic prompt optimization cannot close. Furthermore, we demonstrate a direct correlation between the quality of input requirements and LLM outputs. Our work paves the way to empower more end-users to build complex LLM applications. |
| title | What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use |
| topic | Human-Computer Interaction Artificial Intelligence |
| url | https://arxiv.org/abs/2409.08775 |