Beyond Random Sampling: Instance Quality-Based Data Partitioning via Item Response Theory
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918125116588032 |
|---|---|
| author | Cardoso, Lucas Santos, Vitor Filho, José Ribeiro Prudêncio, Ricardo Kawasaki, Regiane Alves, Ronnie |
| author_facet | Cardoso, Lucas Santos, Vitor Filho, José Ribeiro Prudêncio, Ricardo Kawasaki, Regiane Alves, Ronnie |
| contents | Robust validation of Machine Learning (ML) models is essential, but traditional data partitioning approaches often ignore the intrinsic quality of each instance. This study proposes the use of Item Response Theory (IRT) parameters to characterize and guide the partitioning of datasets in the model validation stage. The impact of IRT-informed partitioning strategies on the performance of several ML models in four tabular datasets was evaluated. The results obtained demonstrate that IRT reveals an inherent heterogeneity of the instances and highlights the existence of informative subgroups of instances within the same dataset. Based on IRT, balanced partitions were created that consistently help to better understand the tradeoff between bias and variance of the models. In addition, the guessing parameter proved to be a determining factor: training with high-guessing instances can significantly impair model performance and resulted in cases with accuracy below 50%, while other partitions reached more than 70% in the same dataset. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_10628 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond Random Sampling: Instance Quality-Based Data Partitioning via Item Response Theory Cardoso, Lucas Santos, Vitor Filho, José Ribeiro Prudêncio, Ricardo Kawasaki, Regiane Alves, Ronnie Machine Learning I.2.6 Robust validation of Machine Learning (ML) models is essential, but traditional data partitioning approaches often ignore the intrinsic quality of each instance. This study proposes the use of Item Response Theory (IRT) parameters to characterize and guide the partitioning of datasets in the model validation stage. The impact of IRT-informed partitioning strategies on the performance of several ML models in four tabular datasets was evaluated. The results obtained demonstrate that IRT reveals an inherent heterogeneity of the instances and highlights the existence of informative subgroups of instances within the same dataset. Based on IRT, balanced partitions were created that consistently help to better understand the tradeoff between bias and variance of the models. In addition, the guessing parameter proved to be a determining factor: training with high-guessing instances can significantly impair model performance and resulted in cases with accuracy below 50%, while other partitions reached more than 70% in the same dataset. |
| title | Beyond Random Sampling: Instance Quality-Based Data Partitioning via Item Response Theory |
| topic | Machine Learning I.2.6 |
| url | https://arxiv.org/abs/2508.10628 |