Salvato in:
| Autore principale: | |
|---|---|
| Natura: | Recurso digital |
| Lingua: | |
| Pubblicazione: |
Zenodo
2025
|
| Accesso online: | https://doi.org/10.5281/zenodo.15532596 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
Sommario:
- <p><strong><em><span lang="EN-US">Abstract</span></em></strong><strong><span lang="EN-US">— Most machine learning models are not only highly dependent on difficult datasets but also on the quality of labeled data they are trained on, especially for offensive content detection. In this paper, we study the TweetEval dataset to provide a comparison of its ground truth with manually annotated labels; inter-annotator agreements are applied here as a metric for assessing the consistency of annotation. Cohen’s Kappa coefficient is used to quantify how much each pair of annotators agreed and where<span> </span>they<span> </span>differed.<span> </span>In-depth<span> </span>examination<span> </span>of<span> </span>missed<span> </span>classifications demonstrates other difficulties with manual labelling: subjective interpretation, context dependency, and annotator bias. The in- sights gathered demonstrate how manual annotation can have positive and negative effects on further model training practices, highlighting<span> </span>the<span> </span>importance<span> </span>of<span> </span>standardized<span> </span>annotation<span> </span>guidelines. In their actions, the findings contribute to enhancing offensive content detection models by advocating dataset reliability and the reduction of inconsistencies in labeling.</span></strong></p> <p><strong><em><span lang="EN-US">Index Terms</span></em></strong><strong><span lang="EN-US">—TweetEval Dataset,Annotation Consistency, Inter- </span></strong><strong><span lang="EN-US">Annotator Agreement,Cohen’s Kappa, Fleiss’ Kappa,Dataset Re- liability, Text Classification,Natural Language Processing (NLP), Offensive Language Detection, Hybrid Models,Annotator Bias</span></strong></p>