How Annotation Trains Annotators: Competence Development in Social Influence Recognition
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917382825443328 |
|---|---|
| author | Markiewicz, Maciej Bajcar, Beata Mieleszczenko-Kowszewicz, Wiktoria Szczęsny, Aleksander Adamczyk, Tomasz Chodak, Grzegorz Ostrowska, Karolina Sawczuk, Aleksandra Babiak, Jolanta Szklarczyk, Jagoda Kazienko, Przemysław |
| author_facet | Markiewicz, Maciej Bajcar, Beata Mieleszczenko-Kowszewicz, Wiktoria Szczęsny, Aleksander Adamczyk, Tomasz Chodak, Grzegorz Ostrowska, Karolina Sawczuk, Aleksandra Babiak, Jolanta Szklarczyk, Jagoda Kazienko, Przemysław |
| contents | Human data annotation, especially when involving experts, is often treated as an objective reference. However, many annotation tasks are inherently subjective, and annotators' judgments may evolve over time. This study investigates changes in the quality of annotators' work from a competence perspective during a process of social influence recognition. The study involved 25 annotators from five different groups, including both experts and non-experts, who annotated a dataset of 1,021 dialogues with 20 social influence techniques, along with intentions, reactions, and consequences. An initial subset of 150 texts was annotated twice - before and after the main annotation process - to enable comparison. To measure competence shifts, we combined qualitative and quantitative analyses of the annotated data, semi-structured interviews with annotators, self-assessment surveys, and Large Language Model training and evaluation on the comparison dataset. The results indicate a significant increase in annotators' self-perceived competence and confidence. Moreover, observed changes in data quality suggest that the annotation process may enhance annotator competence and that this effect is more pronounced in expert groups. The observed shifts in annotator competence have a visible impact on the performance of LLMs trained on their annotated data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_02951 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | How Annotation Trains Annotators: Competence Development in Social Influence Recognition Markiewicz, Maciej Bajcar, Beata Mieleszczenko-Kowszewicz, Wiktoria Szczęsny, Aleksander Adamczyk, Tomasz Chodak, Grzegorz Ostrowska, Karolina Sawczuk, Aleksandra Babiak, Jolanta Szklarczyk, Jagoda Kazienko, Przemysław Computation and Language Artificial Intelligence Human data annotation, especially when involving experts, is often treated as an objective reference. However, many annotation tasks are inherently subjective, and annotators' judgments may evolve over time. This study investigates changes in the quality of annotators' work from a competence perspective during a process of social influence recognition. The study involved 25 annotators from five different groups, including both experts and non-experts, who annotated a dataset of 1,021 dialogues with 20 social influence techniques, along with intentions, reactions, and consequences. An initial subset of 150 texts was annotated twice - before and after the main annotation process - to enable comparison. To measure competence shifts, we combined qualitative and quantitative analyses of the annotated data, semi-structured interviews with annotators, self-assessment surveys, and Large Language Model training and evaluation on the comparison dataset. The results indicate a significant increase in annotators' self-perceived competence and confidence. Moreover, observed changes in data quality suggest that the annotation process may enhance annotator competence and that this effect is more pronounced in expert groups. The observed shifts in annotator competence have a visible impact on the performance of LLMs trained on their annotated data. |
| title | How Annotation Trains Annotators: Competence Development in Social Influence Recognition |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2604.02951 |