Standing on the shoulders of giants
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909307279245312 |
|---|---|
| author | Cardoso, Lucas Felipe Ferraro Filho, José de Sousa Ribeiro Santos, Vitor Cirilo Araujo Frances, Regiane Silva Kawasaki Alves, Ronnie Cley de Oliveira |
| author_facet | Cardoso, Lucas Felipe Ferraro Filho, José de Sousa Ribeiro Santos, Vitor Cirilo Araujo Frances, Regiane Silva Kawasaki Alves, Ronnie Cley de Oliveira |
| contents | Although fundamental to the advancement of Machine Learning, the classic evaluation metrics extracted from the confusion matrix, such as precision and F1, are limited. Such metrics only offer a quantitative view of the models' performance, without considering the complexity of the data or the quality of the hit. To overcome these limitations, recent research has introduced the use of psychometric metrics such as Item Response Theory (IRT), which allows an assessment at the level of latent characteristics of instances. This work investigates how IRT concepts can enrich a confusion matrix in order to identify which model is the most appropriate among options with similar performance. In the study carried out, IRT does not replace, but complements classical metrics by offering a new layer of evaluation and observation of the fine behavior of models in specific instances. It was also observed that there is 97% confidence that the score from the IRT has different contributions from 66% of the classical metrics analyzed. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_03151 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Standing on the shoulders of giants Cardoso, Lucas Felipe Ferraro Filho, José de Sousa Ribeiro Santos, Vitor Cirilo Araujo Frances, Regiane Silva Kawasaki Alves, Ronnie Cley de Oliveira Machine Learning I.2.6 Although fundamental to the advancement of Machine Learning, the classic evaluation metrics extracted from the confusion matrix, such as precision and F1, are limited. Such metrics only offer a quantitative view of the models' performance, without considering the complexity of the data or the quality of the hit. To overcome these limitations, recent research has introduced the use of psychometric metrics such as Item Response Theory (IRT), which allows an assessment at the level of latent characteristics of instances. This work investigates how IRT concepts can enrich a confusion matrix in order to identify which model is the most appropriate among options with similar performance. In the study carried out, IRT does not replace, but complements classical metrics by offering a new layer of evaluation and observation of the fine behavior of models in specific instances. It was also observed that there is 97% confidence that the score from the IRT has different contributions from 66% of the classical metrics analyzed. |
| title | Standing on the shoulders of giants |
| topic | Machine Learning I.2.6 |
| url | https://arxiv.org/abs/2409.03151 |