The Effect of Education in Prompt Engineering: Evidence from Journalists
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917779532152832 |
|---|---|
| author | Bashardoust, Amirsiavosh Feng, Yuanjun Geissler, Dominique Feuerriegel, Stefan Shrestha, Yash Raj |
| author_facet | Bashardoust, Amirsiavosh Feng, Yuanjun Geissler, Dominique Feuerriegel, Stefan Shrestha, Yash Raj |
| contents | Large language models (LLMs) are increasingly used in daily work. In this paper, we analyze whether training in prompt engineering can improve the interactions of users with LLMs. For this, we conducted a field experiment where we asked journalists to write short texts before and after training in prompt engineering. We then analyzed the effect of training on three dimensions: (1) the user experience of journalists when interacting with LLMs, (2) the accuracy of the texts (assessed by a domain expert), and (3) the reader perception, such as clarity, engagement, and other text quality dimensions (assessed by non-expert readers). Our results show: (1) Our training improved the perceived expertise of journalists but also decreased the perceived helpfulness of LLM use. (2) The effect on accuracy varied by the difficulty of the task. (3) There is a mixed impact of training on reader perception across different text quality dimensions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_12320 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Effect of Education in Prompt Engineering: Evidence from Journalists Bashardoust, Amirsiavosh Feng, Yuanjun Geissler, Dominique Feuerriegel, Stefan Shrestha, Yash Raj Human-Computer Interaction Large language models (LLMs) are increasingly used in daily work. In this paper, we analyze whether training in prompt engineering can improve the interactions of users with LLMs. For this, we conducted a field experiment where we asked journalists to write short texts before and after training in prompt engineering. We then analyzed the effect of training on three dimensions: (1) the user experience of journalists when interacting with LLMs, (2) the accuracy of the texts (assessed by a domain expert), and (3) the reader perception, such as clarity, engagement, and other text quality dimensions (assessed by non-expert readers). Our results show: (1) Our training improved the perceived expertise of journalists but also decreased the perceived helpfulness of LLM use. (2) The effect on accuracy varied by the difficulty of the task. (3) There is a mixed impact of training on reader perception across different text quality dimensions. |
| title | The Effect of Education in Prompt Engineering: Evidence from Journalists |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2409.12320 |