Algorithmic Accountability in Small Data: Sample-Size-Induced Bias Within Classification Metrics
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918012136718336 |
|---|---|
| author | Briscoe, Jarren Kepler, Garrett Deford, Daryl Gebremedhin, Assefaw |
| author_facet | Briscoe, Jarren Kepler, Garrett Deford, Daryl Gebremedhin, Assefaw |
| contents | Evaluating machine learning models is crucial not only for determining their technical accuracy but also for assessing their potential societal implications. While the potential for low-sample-size bias in algorithms is well known, we demonstrate the significance of sample-size bias induced by combinatorics in classification metrics. This revelation challenges the efficacy of these metrics in assessing bias with high resolution, especially when comparing groups of disparate sizes, which frequently arise in social applications. We provide analyses of the bias that appears in several commonly applied metrics and propose a model-agnostic assessment and correction technique. Additionally, we analyze counts of undefined cases in metric calculations, which can lead to misleading evaluations if improperly handled. This work illuminates the previously unrecognized challenge of combinatorics and probability in standard evaluation practices and thereby advances approaches for performing fair and trustworthy classification methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_03992 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Algorithmic Accountability in Small Data: Sample-Size-Induced Bias Within Classification Metrics Briscoe, Jarren Kepler, Garrett Deford, Daryl Gebremedhin, Assefaw Machine Learning Evaluating machine learning models is crucial not only for determining their technical accuracy but also for assessing their potential societal implications. While the potential for low-sample-size bias in algorithms is well known, we demonstrate the significance of sample-size bias induced by combinatorics in classification metrics. This revelation challenges the efficacy of these metrics in assessing bias with high resolution, especially when comparing groups of disparate sizes, which frequently arise in social applications. We provide analyses of the bias that appears in several commonly applied metrics and propose a model-agnostic assessment and correction technique. Additionally, we analyze counts of undefined cases in metric calculations, which can lead to misleading evaluations if improperly handled. This work illuminates the previously unrecognized challenge of combinatorics and probability in standard evaluation practices and thereby advances approaches for performing fair and trustworthy classification methods. |
| title | Algorithmic Accountability in Small Data: Sample-Size-Induced Bias Within Classification Metrics |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2505.03992 |