Statistical Mechanics of Support Vector Regression
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916816404611072 |
|---|---|
| author | Canatar, Abdulkadir Chung, SueYeon |
| author_facet | Canatar, Abdulkadir Chung, SueYeon |
| contents | A key problem in deep learning and computational neuroscience is relating the geometrical properties of neural representations to task performance. Here, we consider this problem for continuous decoding tasks where neural variability may affect task precision. Using methods from statistical mechanics, we study the average-case learning curves for $\varepsilon$-insensitive Support Vector Regression ($\varepsilon$-SVR) and discuss its capacity as a measure of linear decodability. Our analysis reveals a phase transition in training error at a critical load, capturing the interplay between the tolerance parameter $\varepsilon$ and neural variability. We uncover a double-descent phenomenon in the generalization error, showing that $\varepsilon$ acts as a regularizer, both suppressing and shifting these peaks. Theoretical predictions are validated both with toy models and deep neural networks, extending the theory of Support Vector Machines to continuous tasks with inherent neural variability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_05439 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Statistical Mechanics of Support Vector Regression Canatar, Abdulkadir Chung, SueYeon Disordered Systems and Neural Networks Neurons and Cognition Machine Learning A key problem in deep learning and computational neuroscience is relating the geometrical properties of neural representations to task performance. Here, we consider this problem for continuous decoding tasks where neural variability may affect task precision. Using methods from statistical mechanics, we study the average-case learning curves for $\varepsilon$-insensitive Support Vector Regression ($\varepsilon$-SVR) and discuss its capacity as a measure of linear decodability. Our analysis reveals a phase transition in training error at a critical load, capturing the interplay between the tolerance parameter $\varepsilon$ and neural variability. We uncover a double-descent phenomenon in the generalization error, showing that $\varepsilon$ acts as a regularizer, both suppressing and shifting these peaks. Theoretical predictions are validated both with toy models and deep neural networks, extending the theory of Support Vector Machines to continuous tasks with inherent neural variability. |
| title | Statistical Mechanics of Support Vector Regression |
| topic | Disordered Systems and Neural Networks Neurons and Cognition Machine Learning |
| url | https://arxiv.org/abs/2412.05439 |