Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.05296 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916473400721408 |
|---|---|
| author | Sohail, Shairoz |
| author_facet | Sohail, Shairoz |
| contents | Kolmogorov-Arnold Networks have recently been introduced as a flexible alternative to multi-layer Perceptron architectures. In this paper, we examine the training dynamics of different KAN architectures and compare them with corresponding MLP formulations. We train with a variety of different initialization schemes, optimizers, and learning rates, as well as utilize back propagation free approaches like the HSIC Bottleneck. We find that (when judged by test accuracy) KANs are an effective alternative to MLP architectures on high-dimensional datasets and have somewhat better parameter efficiency, but suffer from more unstable training dynamics. Finally, we provide recommendations for improving training stability of larger KAN models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_05296 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On Training of Kolmogorov-Arnold Networks Sohail, Shairoz Machine Learning Artificial Intelligence I.2.4 Kolmogorov-Arnold Networks have recently been introduced as a flexible alternative to multi-layer Perceptron architectures. In this paper, we examine the training dynamics of different KAN architectures and compare them with corresponding MLP formulations. We train with a variety of different initialization schemes, optimizers, and learning rates, as well as utilize back propagation free approaches like the HSIC Bottleneck. We find that (when judged by test accuracy) KANs are an effective alternative to MLP architectures on high-dimensional datasets and have somewhat better parameter efficiency, but suffer from more unstable training dynamics. Finally, we provide recommendations for improving training stability of larger KAN models. |
| title | On Training of Kolmogorov-Arnold Networks |
| topic | Machine Learning Artificial Intelligence I.2.4 |
| url | https://arxiv.org/abs/2411.05296 |