Liberal-Conservative Hierarchies of Intercoder Reliability Estimators
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917819144208384 |
|---|---|
| author | Zhao, Yingjie Jay Feng, Guangchao Charles Li, Dianshi Moses Ao, Song Harris Li, Ming Milano Tuo, Zhan Thor Huang, Hui Deng, Ke Zhao, Xinshu |
| author_facet | Zhao, Yingjie Jay Feng, Guangchao Charles Li, Dianshi Moses Ao, Song Harris Li, Ming Milano Tuo, Zhan Thor Huang, Hui Deng, Ke Zhao, Xinshu |
| contents | While numerous indices of inter-coder reliability exist, Krippendorff's α and Cohen's \{kappa} have long dominated in communication studies and other fields, respectively. The near consensus, however, may be near the end. Recent theoretical and mathematical analyses reveal that these indices assume intentional and maximal random coding, leading to paradoxes and inaccuracies. A controlled experiment with one-way golden standard and Monte Carlo simulations supports these findings, showing that \{kappa} and α are poor predictors and approximators of true intercoder reliability. As consensus on a perfect index remains elusive, more authors recommend selecting the best available index for specific situations (BAFS). To make informed choices, researchers, reviewers, and educators need to understand the liberal-conservative hierarchy of indices, i.e., which indices produce higher or lower scores. This study extends previous efforts by expanding the math-based hierarchies to include 23 indices and constructing six additional hierarchies using Monte Carlo simulations. These simulations account for factors like the number of categories and distribution skew. The resulting eight hierarchies display a consistent pattern and reveal a previously undetected paradox in the Ir index. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_05291 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Liberal-Conservative Hierarchies of Intercoder Reliability Estimators Zhao, Yingjie Jay Feng, Guangchao Charles Li, Dianshi Moses Ao, Song Harris Li, Ming Milano Tuo, Zhan Thor Huang, Hui Deng, Ke Zhao, Xinshu Physics and Society While numerous indices of inter-coder reliability exist, Krippendorff's α and Cohen's \{kappa} have long dominated in communication studies and other fields, respectively. The near consensus, however, may be near the end. Recent theoretical and mathematical analyses reveal that these indices assume intentional and maximal random coding, leading to paradoxes and inaccuracies. A controlled experiment with one-way golden standard and Monte Carlo simulations supports these findings, showing that \{kappa} and α are poor predictors and approximators of true intercoder reliability. As consensus on a perfect index remains elusive, more authors recommend selecting the best available index for specific situations (BAFS). To make informed choices, researchers, reviewers, and educators need to understand the liberal-conservative hierarchy of indices, i.e., which indices produce higher or lower scores. This study extends previous efforts by expanding the math-based hierarchies to include 23 indices and constructing six additional hierarchies using Monte Carlo simulations. These simulations account for factors like the number of categories and distribution skew. The resulting eight hierarchies display a consistent pattern and reveal a previously undetected paradox in the Ir index. |
| title | Liberal-Conservative Hierarchies of Intercoder Reliability Estimators |
| topic | Physics and Society |
| url | https://arxiv.org/abs/2410.05291 |