Finite sample rates for logistic regression with small noise or few samples
Fuente:
arXiv
Guardado en:
| Autores principales: | , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916141838893056 |
|---|---|
| author | Kuchelmeister, Felix van de Geer, Sara |
| author_facet | Kuchelmeister, Felix van de Geer, Sara |
| contents | The logistic regression estimator is known to inflate the magnitude of its coefficients if the sample size $n$ is small, the dimension $p$ is (moderately) large or the signal-to-noise ratio $1/σ$ is large (probabilities of observing a label are close to 0 or 1). With this in mind, we study the logistic regression estimator with $p\ll n/\log n$, assuming Gaussian covariates and labels generated by the Gaussian link function, with a mild optimization constraint on the estimator's length to ensure existence. We provide finite sample guarantees for its direction, which serves as a classifier, and its Euclidean norm, which is an estimator for the signal-to-noise ratio. We distinguish between two regimes. In the low-noise/small-sample regime ($σ\lesssim (p\log n)/n$), we show that the estimator's direction (and consequentially the classification error) achieve the rate $(p\log n)/n$ - up to the log term as if the problem was noiseless. In this case, the norm of the estimator is at least of order $n/(p\log n)$. If instead $(p\log n)/n\lesssim σ\lesssim 1$, the estimator's direction achieves the rate $\sqrt{σp\log n/n}$, whereas its norm converges to the true norm at the rate $\sqrt{p\log n/(nσ^3)}$. As a corollary, the data are not linearly separable with high probability in this regime. In either regime, logistic regression provides a competitive classifier. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_15991 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Finite sample rates for logistic regression with small noise or few samples Kuchelmeister, Felix van de Geer, Sara Statistics Theory The logistic regression estimator is known to inflate the magnitude of its coefficients if the sample size $n$ is small, the dimension $p$ is (moderately) large or the signal-to-noise ratio $1/σ$ is large (probabilities of observing a label are close to 0 or 1). With this in mind, we study the logistic regression estimator with $p\ll n/\log n$, assuming Gaussian covariates and labels generated by the Gaussian link function, with a mild optimization constraint on the estimator's length to ensure existence. We provide finite sample guarantees for its direction, which serves as a classifier, and its Euclidean norm, which is an estimator for the signal-to-noise ratio. We distinguish between two regimes. In the low-noise/small-sample regime ($σ\lesssim (p\log n)/n$), we show that the estimator's direction (and consequentially the classification error) achieve the rate $(p\log n)/n$ - up to the log term as if the problem was noiseless. In this case, the norm of the estimator is at least of order $n/(p\log n)$. If instead $(p\log n)/n\lesssim σ\lesssim 1$, the estimator's direction achieves the rate $\sqrt{σp\log n/n}$, whereas its norm converges to the true norm at the rate $\sqrt{p\log n/(nσ^3)}$. As a corollary, the data are not linearly separable with high probability in this regime. In either regime, logistic regression provides a competitive classifier. |
| title | Finite sample rates for logistic regression with small noise or few samples |
| topic | Statistics Theory |
| url | https://arxiv.org/abs/2305.15991 |