Gradient Descent on Logistic Regression: Do Large Step-Sizes Work with Data on the Sphere?
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912484192944128 |
|---|---|
| author | Meng, Si Yi Goujaud, Baptiste Orvieto, Antonio De Sa, Christopher |
| author_facet | Meng, Si Yi Goujaud, Baptiste Orvieto, Antonio De Sa, Christopher |
| contents | Gradient descent (GD) on logistic regression has many fascinating properties. When the dataset is linearly separable, it is known that the iterates converge in direction to the maximum-margin separator regardless of how large the step size is. In the non-separable case, however, it has been shown that GD can exhibit a cycling behaviour even when the step sizes is still below the stability threshold $2/λ$, where $λ$ is the largest eigenvalue of the Hessian at the solution. This short paper explores whether restricting the data to have equal magnitude is a sufficient condition for global convergence, under any step size below the stability threshold. We prove that this is true in a one dimensional space, but in higher dimensions cycling behaviour can still occur. We hope to inspire further studies on quantifying how common these cycles are in realistic datasets, as well as finding sufficient conditions to guarantee global convergence with large step sizes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_11228 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Gradient Descent on Logistic Regression: Do Large Step-Sizes Work with Data on the Sphere? Meng, Si Yi Goujaud, Baptiste Orvieto, Antonio De Sa, Christopher Machine Learning Optimization and Control Gradient descent (GD) on logistic regression has many fascinating properties. When the dataset is linearly separable, it is known that the iterates converge in direction to the maximum-margin separator regardless of how large the step size is. In the non-separable case, however, it has been shown that GD can exhibit a cycling behaviour even when the step sizes is still below the stability threshold $2/λ$, where $λ$ is the largest eigenvalue of the Hessian at the solution. This short paper explores whether restricting the data to have equal magnitude is a sufficient condition for global convergence, under any step size below the stability threshold. We prove that this is true in a one dimensional space, but in higher dimensions cycling behaviour can still occur. We hope to inspire further studies on quantifying how common these cycles are in realistic datasets, as well as finding sufficient conditions to guarantee global convergence with large step sizes. |
| title | Gradient Descent on Logistic Regression: Do Large Step-Sizes Work with Data on the Sphere? |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2507.11228 |