The Positivity of the Neural Tangent Kernel
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914761892954112 |
|---|---|
| author | Carvalho, Luís Costa, João L. Mourão, José Oliveira, Gonçalo |
| author_facet | Carvalho, Luís Costa, João L. Mourão, José Oliveira, Gonçalo |
| contents | The Neural Tangent Kernel (NTK) has emerged as a fundamental concept in the study of wide Neural Networks. In particular, it is known that the positivity of the NTK is directly related to the memorization capacity of sufficiently wide networks, i.e., to the possibility of reaching zero loss in training, via gradient descent. Here we will improve on previous works and obtain a sharp result concerning the positivity of the NTK of feedforward networks of any depth. More precisely, we will show that, for any non-polynomial activation function, the NTK is strictly positive definite. Our results are based on a novel characterization of polynomial functions which is of independent interest. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_12928 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Positivity of the Neural Tangent Kernel Carvalho, Luís Costa, João L. Mourão, José Oliveira, Gonçalo Machine Learning Artificial Intelligence Probability Spectral Theory 68T07, 68R01 The Neural Tangent Kernel (NTK) has emerged as a fundamental concept in the study of wide Neural Networks. In particular, it is known that the positivity of the NTK is directly related to the memorization capacity of sufficiently wide networks, i.e., to the possibility of reaching zero loss in training, via gradient descent. Here we will improve on previous works and obtain a sharp result concerning the positivity of the NTK of feedforward networks of any depth. More precisely, we will show that, for any non-polynomial activation function, the NTK is strictly positive definite. Our results are based on a novel characterization of polynomial functions which is of independent interest. |
| title | The Positivity of the Neural Tangent Kernel |
| topic | Machine Learning Artificial Intelligence Probability Spectral Theory 68T07, 68R01 |
| url | https://arxiv.org/abs/2404.12928 |