From tetrachoric to kappa: How to assess reliability on binary scales
Fuente:
Wiley Open Access
Gespeichert in:
| 1. Verfasser: | |
|---|---|
| Format: | Artículo Open Access |
| Veröffentlicht: |
Wiley
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1867017473500381184 |
|---|---|
| author | Sophie Vanbelle |
| author_facet | Sophie Vanbelle Sophie Vanbelle |
| collection | Wiley Open Access |
| contents | From tetrachoric to kappa: How to assess reliability on binary scales Sophie Vanbelle British Journal of Mathematical and Statistical Psychology Abstract Reliability is crucial in psychometrics, reflecting the extent to which a measurement instrument can discriminate between individuals or items. While classical test theory and intraclass correlation coefficients are well‐established for quantitative scales, estimating reliability for binary outcomes presents unique challenges due to their discrete nature. This paper reviews and links three major approaches to estimate reliability for single ratings on binary scales: the normal approximation approach, kappa coefficients, and the latent variable approach, which enables estimation at both latent and manifest scale levels. We clarify their conceptual relationships, show conditions for asymptotical equivalence, and evaluate their performance across two common study designs, repeatability and reproducibility studies. Then, we extend the Bayesian Dirichlet‐multinomial method for estimating kappa coefficients to settings with more than two replicates, without requiring Bayesian software. Additionally, we introduce a Bayesian method to estimate manifest scale reliability from latent scale reliability that can be implemented in standard Bayesian software. A simulation study compares the statistical properties of the three major approaches across Bayesian and frequentist frameworks. Overall, the normal approximation approach performed poorly, and the frequentist approach was unreliable due to singularity issues. The findings offer further refined practical recommendations. 10.1111/bmsp.70021 http://creativecommons.org/licenses/by/4.0/ |
| doi_str_mv | 10.1111/bmsp.70021 |
| format | Artículo Open Access |
| id | wiley_oa_10_1111_bmsp_70021 |
| institution | Wiley Open Access |
| license_str_mv | http://creativecommons.org/licenses/by/4.0/ |
| publishDate | 2025 |
| publisher | Wiley |
| record_format | wiley_oa |
| spellingShingle | From tetrachoric to kappa: How to assess reliability on binary scales Sophie Vanbelle British Journal of Mathematical and Statistical Psychology From tetrachoric to kappa: How to assess reliability on binary scales Sophie Vanbelle British Journal of Mathematical and Statistical Psychology Abstract Reliability is crucial in psychometrics, reflecting the extent to which a measurement instrument can discriminate between individuals or items. While classical test theory and intraclass correlation coefficients are well‐established for quantitative scales, estimating reliability for binary outcomes presents unique challenges due to their discrete nature. This paper reviews and links three major approaches to estimate reliability for single ratings on binary scales: the normal approximation approach, kappa coefficients, and the latent variable approach, which enables estimation at both latent and manifest scale levels. We clarify their conceptual relationships, show conditions for asymptotical equivalence, and evaluate their performance across two common study designs, repeatability and reproducibility studies. Then, we extend the Bayesian Dirichlet‐multinomial method for estimating kappa coefficients to settings with more than two replicates, without requiring Bayesian software. Additionally, we introduce a Bayesian method to estimate manifest scale reliability from latent scale reliability that can be implemented in standard Bayesian software. A simulation study compares the statistical properties of the three major approaches across Bayesian and frequentist frameworks. Overall, the normal approximation approach performed poorly, and the frequentist approach was unreliable due to singularity issues. The findings offer further refined practical recommendations. 10.1111/bmsp.70021 http://creativecommons.org/licenses/by/4.0/ |
| title | From tetrachoric to kappa: How to assess reliability on binary scales |
| topic | British Journal of Mathematical and Statistical Psychology |
| url | https://bpspsychub.onlinelibrary.wiley.com/doi/10.1111/bmsp.70021 |