Generalization Error Curves for Analytic Spectral Algorithms under Power-law Decay
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915031222845440 |
|---|---|
| author | Li, Yicheng Gan, Weiye Shi, Zuoqiang Lin, Qian |
| author_facet | Li, Yicheng Gan, Weiye Shi, Zuoqiang Lin, Qian |
| contents | The generalization error curve of certain kernel regression method aims at determining the exact order of generalization error with various source condition, noise level and choice of the regularization parameter rather than the minimax rate. In this work, under mild assumptions, we rigorously provide a full characterization of the generalization error curves of the kernel gradient descent method (and a large class of analytic spectral algorithms) in kernel regression. Consequently, we could sharpen the near inconsistency of kernel interpolation and clarify the saturation effects of kernel regression algorithms with higher qualification, etc. Thanks to the neural tangent kernel theory, these results greatly improve our understanding of the generalization behavior of training the wide neural networks. A novel technical contribution, the analytic functional argument, might be of independent interest. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_01599 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Generalization Error Curves for Analytic Spectral Algorithms under Power-law Decay Li, Yicheng Gan, Weiye Shi, Zuoqiang Lin, Qian Machine Learning Statistics Theory The generalization error curve of certain kernel regression method aims at determining the exact order of generalization error with various source condition, noise level and choice of the regularization parameter rather than the minimax rate. In this work, under mild assumptions, we rigorously provide a full characterization of the generalization error curves of the kernel gradient descent method (and a large class of analytic spectral algorithms) in kernel regression. Consequently, we could sharpen the near inconsistency of kernel interpolation and clarify the saturation effects of kernel regression algorithms with higher qualification, etc. Thanks to the neural tangent kernel theory, these results greatly improve our understanding of the generalization behavior of training the wide neural networks. A novel technical contribution, the analytic functional argument, might be of independent interest. |
| title | Generalization Error Curves for Analytic Spectral Algorithms under Power-law Decay |
| topic | Machine Learning Statistics Theory |
| url | https://arxiv.org/abs/2401.01599 |