Enhancing Neural Training via a Correlated Dynamics Model
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929431047569408 |
|---|---|
| author | Brokman, Jonathan Betser, Roy Turjeman, Rotem Berkov, Tom Cohen, Ido Gilboa, Guy |
| author_facet | Brokman, Jonathan Betser, Roy Turjeman, Rotem Berkov, Tom Cohen, Ido Gilboa, Guy |
| contents | As neural networks grow in scale, their training becomes both computationally demanding and rich in dynamics. Amidst the flourishing interest in these training dynamics, we present a novel observation: Parameters during training exhibit intrinsic correlations over time. Capitalizing on this, we introduce Correlation Mode Decomposition (CMD). This algorithm clusters the parameter space into groups, termed modes, that display synchronized behavior across epochs. This enables CMD to efficiently represent the training dynamics of complex networks, like ResNets and Transformers, using only a few modes. Moreover, test set generalization is enhanced. We introduce an efficient CMD variant, designed to run concurrently with training. Our experiments indicate that CMD surpasses the state-of-the-art method for compactly modeled dynamics on image classification. Our modeling can improve training efficiency and lower communication overhead, as shown by our preliminary experiments in the context of federated learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_13247 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Enhancing Neural Training via a Correlated Dynamics Model Brokman, Jonathan Betser, Roy Turjeman, Rotem Berkov, Tom Cohen, Ido Gilboa, Guy Machine Learning Dynamical Systems As neural networks grow in scale, their training becomes both computationally demanding and rich in dynamics. Amidst the flourishing interest in these training dynamics, we present a novel observation: Parameters during training exhibit intrinsic correlations over time. Capitalizing on this, we introduce Correlation Mode Decomposition (CMD). This algorithm clusters the parameter space into groups, termed modes, that display synchronized behavior across epochs. This enables CMD to efficiently represent the training dynamics of complex networks, like ResNets and Transformers, using only a few modes. Moreover, test set generalization is enhanced. We introduce an efficient CMD variant, designed to run concurrently with training. Our experiments indicate that CMD surpasses the state-of-the-art method for compactly modeled dynamics on image classification. Our modeling can improve training efficiency and lower communication overhead, as shown by our preliminary experiments in the context of federated learning. |
| title | Enhancing Neural Training via a Correlated Dynamics Model |
| topic | Machine Learning Dynamical Systems |
| url | https://arxiv.org/abs/2312.13247 |