Neural Velocity for hyperparameter tuning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908439998889984 |
|---|---|
| author | Dalmasso, Gianluca Bragagnolo, Andrea Tartaglione, Enzo Fiandrotti, Attilio Grangetto, Marco |
| author_facet | Dalmasso, Gianluca Bragagnolo, Andrea Tartaglione, Enzo Fiandrotti, Attilio Grangetto, Marco |
| contents | Hyperparameter tuning, such as learning rate decay and defining a stopping criterion, often relies on monitoring the validation loss. This paper presents NeVe, a dynamic training approach that adjusts the learning rate and defines the stop criterion based on the novel notion of "neural velocity". The neural velocity measures the rate of change of each neuron's transfer function and is an indicator of model convergence: sampling neural velocity can be performed even by forwarding noise in the network, reducing the need for a held-out dataset. Our findings show the potential of neural velocity as a key metric for optimizing neural network training efficiently |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_05309 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Neural Velocity for hyperparameter tuning Dalmasso, Gianluca Bragagnolo, Andrea Tartaglione, Enzo Fiandrotti, Attilio Grangetto, Marco Machine Learning Artificial Intelligence 68T05 I.2.6; I.5.1; I.5.4 Hyperparameter tuning, such as learning rate decay and defining a stopping criterion, often relies on monitoring the validation loss. This paper presents NeVe, a dynamic training approach that adjusts the learning rate and defines the stop criterion based on the novel notion of "neural velocity". The neural velocity measures the rate of change of each neuron's transfer function and is an indicator of model convergence: sampling neural velocity can be performed even by forwarding noise in the network, reducing the need for a held-out dataset. Our findings show the potential of neural velocity as a key metric for optimizing neural network training efficiently |
| title | Neural Velocity for hyperparameter tuning |
| topic | Machine Learning Artificial Intelligence 68T05 I.2.6; I.5.1; I.5.4 |
| url | https://arxiv.org/abs/2507.05309 |