Hankel Singular Value Regularization for Highly Compressible State Space Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911241066250240 |
|---|---|
| author | Schwerdtner, Paul Berman, Jules Peherstorfer, Benjamin |
| author_facet | Schwerdtner, Paul Berman, Jules Peherstorfer, Benjamin |
| contents | Deep neural networks using state space models as layers are well suited for long-range sequence tasks but can be challenging to compress after training. We use that regularizing the sum of Hankel singular values of state space models leads to a fast decay of these singular values and thus to compressible models. To make the proposed Hankel singular value regularization scalable, we develop an algorithm to efficiently compute the Hankel singular values during training iterations by exploiting the specific block-diagonal structure of the system matrices that we use in our state space model parametrization. Experiments on Long Range Arena benchmarks demonstrate that the regularized state space layers are up to 10$\times$ more compressible than standard state space layers while maintaining high accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_22951 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Hankel Singular Value Regularization for Highly Compressible State Space Models Schwerdtner, Paul Berman, Jules Peherstorfer, Benjamin Machine Learning Dynamical Systems Deep neural networks using state space models as layers are well suited for long-range sequence tasks but can be challenging to compress after training. We use that regularizing the sum of Hankel singular values of state space models leads to a fast decay of these singular values and thus to compressible models. To make the proposed Hankel singular value regularization scalable, we develop an algorithm to efficiently compute the Hankel singular values during training iterations by exploiting the specific block-diagonal structure of the system matrices that we use in our state space model parametrization. Experiments on Long Range Arena benchmarks demonstrate that the regularized state space layers are up to 10$\times$ more compressible than standard state space layers while maintaining high accuracy. |
| title | Hankel Singular Value Regularization for Highly Compressible State Space Models |
| topic | Machine Learning Dynamical Systems |
| url | https://arxiv.org/abs/2510.22951 |