Analysis of Long Range Dependency Understanding in State Space Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911384885788672 |
|---|---|
| author | Ravikumar, Srividya Anand, Abhinav Verma, Shweta Mezini, Mira |
| author_facet | Ravikumar, Srividya Anand, Abhinav Verma, Shweta Mezini, Mira |
| contents | Although state-space models (SSMs) have demonstrated strong performance on long-sequence benchmarks, most research has emphasized predictive accuracy rather than interpretability. In this work, we present the first systematic kernel interpretability study of the diagonalized state-space model (S4D) trained on a real-world task (vulnerability detection in source code). Through time and frequency domain analysis of the S4D kernel, we show that the long-range modeling capability of S4D varies significantly under different model architectures, affecting model performance. For instance, we show that the depending on the architecture, S4D kernel can behave as low-pass, band-pass or high-pass filter. The insights from our analysis can guide future work in designing better S4D-based models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_13048 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Analysis of Long Range Dependency Understanding in State Space Models Ravikumar, Srividya Anand, Abhinav Verma, Shweta Mezini, Mira Machine Learning Artificial Intelligence Although state-space models (SSMs) have demonstrated strong performance on long-sequence benchmarks, most research has emphasized predictive accuracy rather than interpretability. In this work, we present the first systematic kernel interpretability study of the diagonalized state-space model (S4D) trained on a real-world task (vulnerability detection in source code). Through time and frequency domain analysis of the S4D kernel, we show that the long-range modeling capability of S4D varies significantly under different model architectures, affecting model performance. For instance, we show that the depending on the architecture, S4D kernel can behave as low-pass, band-pass or high-pass filter. The insights from our analysis can guide future work in designing better S4D-based models. |
| title | Analysis of Long Range Dependency Understanding in State Space Models |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2601.13048 |