Prompt Adaptation as a Dynamic Complement in Generative AI Systems
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911357123690496 |
|---|---|
| author | Jahani, Eaman Manning, Benjamin S. Zhang, Joe TuYe, Hong-Yi Alsobay, Mohammed Nicolaides, Christos Suri, Siddharth Holtz, David |
| author_facet | Jahani, Eaman Manning, Benjamin S. Zhang, Joe TuYe, Hong-Yi Alsobay, Mohammed Nicolaides, Christos Suri, Siddharth Holtz, David |
| contents | As generative AI systems rapidly improve, a key question emerges: how do users adapt to these changes, and when does such adaptation matter for realizing performance gains? Drawing on theories of dynamic capabilities and IT complements, we study prompt adaptation--how users adjust their inputs in response to evolving model behavior--using a common experimental design applied to two preregistered tasks with 3,750 total participants who submitted nearly 37,000 prompts. We show that the importance of prompt adaptation depends critically on task structure. In a task with fixed evaluation criteria and an unambiguous goal, user prompt adaptation accounts for roughly half of the performance gains from a model upgrade. In contrast, in an open-ended creative task where the space of acceptable outputs is effectively unbounded and quality is subjective, performance improvements are driven primarily by model capability; prompt adaptation plays a limited role. We further show that automated prompt rewriting cannot generally substitute for human adaptation: when aligned with task objectives, it can modestly improve performance, but when misaligned, it can actively undermine the gains from model improvements. Together, these findings position prompt adaptation as a dynamic complement whose importance depends on task structure and system design, and suggest that without it, a substantial share of the economic value created by advances in generative models may go unrealized. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_14333 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Prompt Adaptation as a Dynamic Complement in Generative AI Systems Jahani, Eaman Manning, Benjamin S. Zhang, Joe TuYe, Hong-Yi Alsobay, Mohammed Nicolaides, Christos Suri, Siddharth Holtz, David Human-Computer Interaction General Economics Economics As generative AI systems rapidly improve, a key question emerges: how do users adapt to these changes, and when does such adaptation matter for realizing performance gains? Drawing on theories of dynamic capabilities and IT complements, we study prompt adaptation--how users adjust their inputs in response to evolving model behavior--using a common experimental design applied to two preregistered tasks with 3,750 total participants who submitted nearly 37,000 prompts. We show that the importance of prompt adaptation depends critically on task structure. In a task with fixed evaluation criteria and an unambiguous goal, user prompt adaptation accounts for roughly half of the performance gains from a model upgrade. In contrast, in an open-ended creative task where the space of acceptable outputs is effectively unbounded and quality is subjective, performance improvements are driven primarily by model capability; prompt adaptation plays a limited role. We further show that automated prompt rewriting cannot generally substitute for human adaptation: when aligned with task objectives, it can modestly improve performance, but when misaligned, it can actively undermine the gains from model improvements. Together, these findings position prompt adaptation as a dynamic complement whose importance depends on task structure and system design, and suggest that without it, a substantial share of the economic value created by advances in generative models may go unrealized. |
| title | Prompt Adaptation as a Dynamic Complement in Generative AI Systems |
| topic | Human-Computer Interaction General Economics Economics |
| url | https://arxiv.org/abs/2407.14333 |