ARTICLE: Annotator Reliability Through In-Context Learning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913509163401216 |
|---|---|
| author | Dutta, Sujan Pandita, Deepak Weerasooriya, Tharindu Cyril Zampieri, Marcos Homan, Christopher M. KhudaBukhsh, Ashiqur R. |
| author_facet | Dutta, Sujan Pandita, Deepak Weerasooriya, Tharindu Cyril Zampieri, Marcos Homan, Christopher M. KhudaBukhsh, Ashiqur R. |
| contents | Ensuring annotator quality in training and evaluation data is a key piece of machine learning in NLP. Tasks such as sentiment analysis and offensive speech detection are intrinsically subjective, creating a challenging scenario for traditional quality assessment approaches because it is hard to distinguish disagreement due to poor work from that due to differences of opinions between sincere annotators. With the goal of increasing diverse perspectives in annotation while ensuring consistency, we propose \texttt{ARTICLE}, an in-context learning (ICL) framework to estimate annotation quality through self-consistency. We evaluate this framework on two offensive speech datasets using multiple LLMs and compare its performance with traditional methods. Our findings indicate that \texttt{ARTICLE} can be used as a robust method for identifying reliable annotators, hence improving data quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_12218 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ARTICLE: Annotator Reliability Through In-Context Learning Dutta, Sujan Pandita, Deepak Weerasooriya, Tharindu Cyril Zampieri, Marcos Homan, Christopher M. KhudaBukhsh, Ashiqur R. Computation and Language Machine Learning Ensuring annotator quality in training and evaluation data is a key piece of machine learning in NLP. Tasks such as sentiment analysis and offensive speech detection are intrinsically subjective, creating a challenging scenario for traditional quality assessment approaches because it is hard to distinguish disagreement due to poor work from that due to differences of opinions between sincere annotators. With the goal of increasing diverse perspectives in annotation while ensuring consistency, we propose \texttt{ARTICLE}, an in-context learning (ICL) framework to estimate annotation quality through self-consistency. We evaluate this framework on two offensive speech datasets using multiple LLMs and compare its performance with traditional methods. Our findings indicate that \texttt{ARTICLE} can be used as a robust method for identifying reliable annotators, hence improving data quality. |
| title | ARTICLE: Annotator Reliability Through In-Context Learning |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2409.12218 |