Derivation of effective gradient flow equations and dynamical truncation of training data in Deep Learning
Fuente:
arXiv
Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915103018844160 |
|---|---|
| author | Chen, Thomas |
| author_facet | Chen, Thomas |
| contents | We derive explicit equations governing the cumulative biases and weights in Deep Learning with ReLU activation function, based on gradient descent for the Euclidean cost in the input layer, and under the assumption that the weights are, in a precise sense, adapted to the coordinate system distinguished by the activations. We show that gradient descent corresponds to a dynamical process in the input layer, whereby clusters of data are progressively reduced in complexity ("truncated") at an exponential rate that increases with the number of data points that have already been truncated. We provide a detailed discussion of several types of solutions to the gradient flow equations. A main motivation for this work is to shed light on the interpretability question in supervised learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_07400 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Derivation of effective gradient flow equations and dynamical truncation of training data in Deep Learning Chen, Thomas Machine Learning Artificial Intelligence Analysis of PDEs Optimization and Control 57R70, 62M45 We derive explicit equations governing the cumulative biases and weights in Deep Learning with ReLU activation function, based on gradient descent for the Euclidean cost in the input layer, and under the assumption that the weights are, in a precise sense, adapted to the coordinate system distinguished by the activations. We show that gradient descent corresponds to a dynamical process in the input layer, whereby clusters of data are progressively reduced in complexity ("truncated") at an exponential rate that increases with the number of data points that have already been truncated. We provide a detailed discussion of several types of solutions to the gradient flow equations. A main motivation for this work is to shed light on the interpretability question in supervised learning. |
| title | Derivation of effective gradient flow equations and dynamical truncation of training data in Deep Learning |
| topic | Machine Learning Artificial Intelligence Analysis of PDEs Optimization and Control 57R70, 62M45 |
| url | https://arxiv.org/abs/2501.07400 |