CoPrompter: User-Centric Evaluation of LLM Instruction Alignment for Improved Prompt Engineering
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917832823930880 |
|---|---|
| author | Joshi, Ishika Shahid, Simra Venneti, Shreeya Vasu, Manushree Zheng, Yantao Li, Yunyao Krishnamurthy, Balaji Chan, Gromit Yeuk-Yin |
| author_facet | Joshi, Ishika Shahid, Simra Venneti, Shreeya Vasu, Manushree Zheng, Yantao Li, Yunyao Krishnamurthy, Balaji Chan, Gromit Yeuk-Yin |
| contents | Ensuring large language models' (LLMs) responses align with prompt instructions is crucial for application development. Based on our formative study with industry professionals, the alignment requires heavy human involvement and tedious trial-and-error especially when there are many instructions in the prompt. To address these challenges, we introduce CoPrompter, a framework that identifies misalignment based on assessing multiple LLM responses with criteria. It proposes a method to generate evaluation criteria questions derived directly from prompt requirements and an interface to turn these questions into a user-editable checklist. Our user study with industry prompt engineers shows that CoPrompter improves the ability to identify and refine instruction alignment with prompt requirements over traditional methods, helps them understand where and how frequently models fail to follow user's prompt requirements, and helps in clarifying their own requirements, giving them greater control over the response evaluation process. We also present the design lessons to underscore our system's potential to streamline the prompt engineering process. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_06099 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CoPrompter: User-Centric Evaluation of LLM Instruction Alignment for Improved Prompt Engineering Joshi, Ishika Shahid, Simra Venneti, Shreeya Vasu, Manushree Zheng, Yantao Li, Yunyao Krishnamurthy, Balaji Chan, Gromit Yeuk-Yin Human-Computer Interaction Ensuring large language models' (LLMs) responses align with prompt instructions is crucial for application development. Based on our formative study with industry professionals, the alignment requires heavy human involvement and tedious trial-and-error especially when there are many instructions in the prompt. To address these challenges, we introduce CoPrompter, a framework that identifies misalignment based on assessing multiple LLM responses with criteria. It proposes a method to generate evaluation criteria questions derived directly from prompt requirements and an interface to turn these questions into a user-editable checklist. Our user study with industry prompt engineers shows that CoPrompter improves the ability to identify and refine instruction alignment with prompt requirements over traditional methods, helps them understand where and how frequently models fail to follow user's prompt requirements, and helps in clarifying their own requirements, giving them greater control over the response evaluation process. We also present the design lessons to underscore our system's potential to streamline the prompt engineering process. |
| title | CoPrompter: User-Centric Evaluation of LLM Instruction Alignment for Improved Prompt Engineering |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2411.06099 |