A Framework for Optimizing Human-Machine Interaction in Classification Systems
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911368092844032 |
|---|---|
| author | Muric, Goran Minton, Steven |
| author_facet | Muric, Goran Minton, Steven |
| contents | Automated decision systems increasingly rely on human oversight to ensure accuracy in uncertain cases. This paper presents a practical framework for optimizing such human-in-the-loop classification systems using a double-threshold policy. Conventional classifiers usually produce a confidence score and apply a single cutoff, but our approach uses two thresholds (a lower and an upper) to automatically accept or reject high-confidence cases while routing ambiguous instances to human reviewers. We formulate this problem as an optimization task that balances system accuracy against the cost of human review. Through analytical derivations and Monte Carlo simulations, we show how different confidence score distributions impact the efficiency of human intervention and reveal regions of diminishing returns, where additional review yields minimal benefit. The framework provides a general, reproducible method for improving reliability in any decision pipeline requiring selective human validation, including applications in entity resolution, fraud detection, medical triage, and content moderation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_05974 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | A Framework for Optimizing Human-Machine Interaction in Classification Systems Muric, Goran Minton, Steven Human-Computer Interaction Automated decision systems increasingly rely on human oversight to ensure accuracy in uncertain cases. This paper presents a practical framework for optimizing such human-in-the-loop classification systems using a double-threshold policy. Conventional classifiers usually produce a confidence score and apply a single cutoff, but our approach uses two thresholds (a lower and an upper) to automatically accept or reject high-confidence cases while routing ambiguous instances to human reviewers. We formulate this problem as an optimization task that balances system accuracy against the cost of human review. Through analytical derivations and Monte Carlo simulations, we show how different confidence score distributions impact the efficiency of human intervention and reveal regions of diminishing returns, where additional review yields minimal benefit. The framework provides a general, reproducible method for improving reliability in any decision pipeline requiring selective human validation, including applications in entity resolution, fraud detection, medical triage, and content moderation. |
| title | A Framework for Optimizing Human-Machine Interaction in Classification Systems |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2601.05974 |