Guardado en:
| Autor principal: | |
|---|---|
| Formato: | Recurso digital |
| Lenguaje: | |
| Publicado: |
Zenodo
2025
|
| Acceso en línea: | https://doi.org/10.5281/zenodo.15532596 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866901660548202496 |
|---|---|
| author | Fabeela Ali Rawther,Abhinay A K,Anagha Tess B,Alan Joseph,Adham Saheer |
| author_facet | Fabeela Ali Rawther,Abhinay A K,Anagha Tess B,Alan Joseph,Adham Saheer |
| contents | <p><strong><em><span lang="EN-US">Abstract</span></em></strong><strong><span lang="EN-US">— Most machine learning models are not only highly dependent on difficult datasets but also on the quality of labeled data they are trained on, especially for offensive content detection. In this paper, we study the TweetEval dataset to provide a comparison of its ground truth with manually annotated labels; inter-annotator agreements are applied here as a metric for assessing the consistency of annotation. Cohen’s Kappa coefficient is used to quantify how much each pair of annotators agreed and where<span> </span>they<span> </span>differed.<span> </span>In-depth<span> </span>examination<span> </span>of<span> </span>missed<span> </span>classifications demonstrates other difficulties with manual labelling: subjective interpretation, context dependency, and annotator bias. The in- sights gathered demonstrate how manual annotation can have positive and negative effects on further model training practices, highlighting<span> </span>the<span> </span>importance<span> </span>of<span> </span>standardized<span> </span>annotation<span> </span>guidelines. In their actions, the findings contribute to enhancing offensive content detection models by advocating dataset reliability and the reduction of inconsistencies in labeling.</span></strong></p> <p><strong><em><span lang="EN-US">Index Terms</span></em></strong><strong><span lang="EN-US">—TweetEval Dataset,Annotation Consistency, Inter- </span></strong><strong><span lang="EN-US">Annotator Agreement,Cohen’s Kappa, Fleiss’ Kappa,Dataset Re- liability, Text Classification,Natural Language Processing (NLP), Offensive Language Detection, Hybrid Models,Annotator Bias</span></strong></p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_15532596 |
| institution | Zenodo |
| language | |
| publishDate | 2025 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | Evaluating Annotation Consistency in Offensive Language Detection: A Data Analytics Approach on the TweetEval Dataset Fabeela Ali Rawther,Abhinay A K,Anagha Tess B,Alan Joseph,Adham Saheer <p><strong><em><span lang="EN-US">Abstract</span></em></strong><strong><span lang="EN-US">— Most machine learning models are not only highly dependent on difficult datasets but also on the quality of labeled data they are trained on, especially for offensive content detection. In this paper, we study the TweetEval dataset to provide a comparison of its ground truth with manually annotated labels; inter-annotator agreements are applied here as a metric for assessing the consistency of annotation. Cohen’s Kappa coefficient is used to quantify how much each pair of annotators agreed and where<span> </span>they<span> </span>differed.<span> </span>In-depth<span> </span>examination<span> </span>of<span> </span>missed<span> </span>classifications demonstrates other difficulties with manual labelling: subjective interpretation, context dependency, and annotator bias. The in- sights gathered demonstrate how manual annotation can have positive and negative effects on further model training practices, highlighting<span> </span>the<span> </span>importance<span> </span>of<span> </span>standardized<span> </span>annotation<span> </span>guidelines. In their actions, the findings contribute to enhancing offensive content detection models by advocating dataset reliability and the reduction of inconsistencies in labeling.</span></strong></p> <p><strong><em><span lang="EN-US">Index Terms</span></em></strong><strong><span lang="EN-US">—TweetEval Dataset,Annotation Consistency, Inter- </span></strong><strong><span lang="EN-US">Annotator Agreement,Cohen’s Kappa, Fleiss’ Kappa,Dataset Re- liability, Text Classification,Natural Language Processing (NLP), Offensive Language Detection, Hybrid Models,Annotator Bias</span></strong></p> |
| title | Evaluating Annotation Consistency in Offensive Language Detection: A Data Analytics Approach on the TweetEval Dataset |
| url | https://doi.org/10.5281/zenodo.15532596 |