Rater Cohesion and Quality from a Vicarious Perspective
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916424333656064 |
|---|---|
| author | Pandita, Deepak Weerasooriya, Tharindu Cyril Dutta, Sujan Luger, Sarah K. Ranasinghe, Tharindu KhudaBukhsh, Ashiqur R. Zampieri, Marcos Homan, Christopher M. |
| author_facet | Pandita, Deepak Weerasooriya, Tharindu Cyril Dutta, Sujan Luger, Sarah K. Ranasinghe, Tharindu KhudaBukhsh, Ashiqur R. Zampieri, Marcos Homan, Christopher M. |
| contents | Human feedback is essential for building human-centered AI systems across domains where disagreement is prevalent, such as AI safety, content moderation, or sentiment analysis. Many disagreements, particularly in politically charged settings, arise because raters have opposing values or beliefs. Vicarious annotation is a method for breaking down disagreement by asking raters how they think others would annotate the data. In this paper, we explore the use of vicarious annotation with analytical methods for moderating rater disagreement. We employ rater cohesion metrics to study the potential influence of political affiliations and demographic backgrounds on raters' perceptions of offense. Additionally, we utilize CrowdTruth's rater quality metrics, which consider the demographics of the raters, to score the raters and their annotations. We study how the rater quality metrics influence the in-group and cross-group rater cohesion across the personal and vicarious levels. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_08411 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Rater Cohesion and Quality from a Vicarious Perspective Pandita, Deepak Weerasooriya, Tharindu Cyril Dutta, Sujan Luger, Sarah K. Ranasinghe, Tharindu KhudaBukhsh, Ashiqur R. Zampieri, Marcos Homan, Christopher M. Computation and Language Human feedback is essential for building human-centered AI systems across domains where disagreement is prevalent, such as AI safety, content moderation, or sentiment analysis. Many disagreements, particularly in politically charged settings, arise because raters have opposing values or beliefs. Vicarious annotation is a method for breaking down disagreement by asking raters how they think others would annotate the data. In this paper, we explore the use of vicarious annotation with analytical methods for moderating rater disagreement. We employ rater cohesion metrics to study the potential influence of political affiliations and demographic backgrounds on raters' perceptions of offense. Additionally, we utilize CrowdTruth's rater quality metrics, which consider the demographics of the raters, to score the raters and their annotations. We study how the rater quality metrics influence the in-group and cross-group rater cohesion across the personal and vicarious levels. |
| title | Rater Cohesion and Quality from a Vicarious Perspective |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2408.08411 |