Evaluating WAIC and PSIS-LOO for Bayesian Diagnostic Classification Model Selection
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913530406502400 |
|---|---|
| author | Jung, Ae Kyong Templin, Jonathan |
| author_facet | Jung, Ae Kyong Templin, Jonathan |
| contents | Bayesian diagnostic classification models (Bayesian DCMs) are effective for diagnosing students' skills. Research on the evaluation of relative model fit indices for DCMs using Bayesian estimation, however, is deficient. This study introduces the performance of Bayesian relative model fit indices, the widely applicable information criterion (WAIC) and leave-one-out cross-validation using Pareto-smoothed importance sampling (PSIS-LOO), in comparison to simpler and more widely used deviance information criterion (DIC). The simulation study evaluates the performance of WAIC and PSIS-LOO by detecting the true model with varying sample sizes, item qualities, and prior information levels. The results of the study indicate that WAIC and PSIS-LOO primarily favored the generating model; however, occasional inconsistencies were observed. This study recommends using WAIC and PSIS-LOO when the data is assumed to follow a simpler model and the models are estimated under uninformative priors, and DIC when the data is assumed to follow a more complex model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_02931 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Evaluating WAIC and PSIS-LOO for Bayesian Diagnostic Classification Model Selection Jung, Ae Kyong Templin, Jonathan Applications Bayesian diagnostic classification models (Bayesian DCMs) are effective for diagnosing students' skills. Research on the evaluation of relative model fit indices for DCMs using Bayesian estimation, however, is deficient. This study introduces the performance of Bayesian relative model fit indices, the widely applicable information criterion (WAIC) and leave-one-out cross-validation using Pareto-smoothed importance sampling (PSIS-LOO), in comparison to simpler and more widely used deviance information criterion (DIC). The simulation study evaluates the performance of WAIC and PSIS-LOO by detecting the true model with varying sample sizes, item qualities, and prior information levels. The results of the study indicate that WAIC and PSIS-LOO primarily favored the generating model; however, occasional inconsistencies were observed. This study recommends using WAIC and PSIS-LOO when the data is assumed to follow a simpler model and the models are estimated under uninformative priors, and DIC when the data is assumed to follow a more complex model. |
| title | Evaluating WAIC and PSIS-LOO for Bayesian Diagnostic Classification Model Selection |
| topic | Applications |
| url | https://arxiv.org/abs/2410.02931 |