Pathological Regularization Regimes in Classification Tasks
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914843552907264 |
|---|---|
| author | Wiesmann, Maximilian Larsen, Paul |
| author_facet | Wiesmann, Maximilian Larsen, Paul |
| contents | In this paper we demonstrate the possibility of a trend reversal in binary classification tasks between the dataset and a classification score obtained from a trained model. This trend reversal occurs for certain choices of the regularization parameter for model training, namely, if the parameter is contained in what we call the pathological regularization regime. For ridge regression, we give necessary and sufficient algebraic conditions on the dataset for the existence of a pathological regularization regime. Moreover, our results provide a data science practitioner with a hands-on tool to avoid hyperparameter choices suffering from trend reversal. We furthermore present numerical results on pathological regularization regimes for logistic regression. Finally, we draw connections to datasets exhibiting Simpson's paradox, providing a natural source of pathological datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_14731 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Pathological Regularization Regimes in Classification Tasks Wiesmann, Maximilian Larsen, Paul Machine Learning Statistics Theory 62J07 (Primary), 62J12, 62R07 (Secondary) In this paper we demonstrate the possibility of a trend reversal in binary classification tasks between the dataset and a classification score obtained from a trained model. This trend reversal occurs for certain choices of the regularization parameter for model training, namely, if the parameter is contained in what we call the pathological regularization regime. For ridge regression, we give necessary and sufficient algebraic conditions on the dataset for the existence of a pathological regularization regime. Moreover, our results provide a data science practitioner with a hands-on tool to avoid hyperparameter choices suffering from trend reversal. We furthermore present numerical results on pathological regularization regimes for logistic regression. Finally, we draw connections to datasets exhibiting Simpson's paradox, providing a natural source of pathological datasets. |
| title | Pathological Regularization Regimes in Classification Tasks |
| topic | Machine Learning Statistics Theory 62J07 (Primary), 62J12, 62R07 (Secondary) |
| url | https://arxiv.org/abs/2406.14731 |