Towards Automating Text Annotation: A Case Study on Semantic Proximity Annotation using GPT-4
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913418051584000 |
|---|---|
| author | Yadav, Sachin Choppa, Tejaswi Schlechtweg, Dominik |
| author_facet | Yadav, Sachin Choppa, Tejaswi Schlechtweg, Dominik |
| contents | This paper explores using GPT-3.5 and GPT-4 to automate the data annotation process with automatic prompting techniques. The main aim of this paper is to reuse human annotation guidelines along with some annotated data to design automatic prompts for LLMs, focusing on the semantic proximity annotation task. Automatic prompts are compared to customized prompts. We further implement the prompting strategies into an open-source text annotation tool, enabling easy online use via the OpenAI API. Our study reveals the crucial role of accurate prompt design and suggests that prompting GPT-4 with human-like instructions is not straightforwardly possible for the semantic proximity task. We show that small modifications to the human guidelines already improve the performance, suggesting possible ways for future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_04130 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Automating Text Annotation: A Case Study on Semantic Proximity Annotation using GPT-4 Yadav, Sachin Choppa, Tejaswi Schlechtweg, Dominik Computation and Language This paper explores using GPT-3.5 and GPT-4 to automate the data annotation process with automatic prompting techniques. The main aim of this paper is to reuse human annotation guidelines along with some annotated data to design automatic prompts for LLMs, focusing on the semantic proximity annotation task. Automatic prompts are compared to customized prompts. We further implement the prompting strategies into an open-source text annotation tool, enabling easy online use via the OpenAI API. Our study reveals the crucial role of accurate prompt design and suggests that prompting GPT-4 with human-like instructions is not straightforwardly possible for the semantic proximity task. We show that small modifications to the human guidelines already improve the performance, suggesting possible ways for future research. |
| title | Towards Automating Text Annotation: A Case Study on Semantic Proximity Annotation using GPT-4 |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2407.04130 |