Leveraging State Space Models in Long Range Genomics
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908359011074048 |
|---|---|
| author | Popov, Matvei Kallala, Aymen Ramesh, Anirudha Hennouni, Narimane Khaitan, Shivesh Gentry, Rick Cohen, Alain-Sam |
| author_facet | Popov, Matvei Kallala, Aymen Ramesh, Anirudha Hennouni, Narimane Khaitan, Shivesh Gentry, Rick Cohen, Alain-Sam |
| contents | Long-range dependencies are critical for understanding genomic structure and function, yet most conventional methods struggle with them. Widely adopted transformer-based models, while excelling at short-context tasks, are limited by the attention module's quadratic computational complexity and inability to extrapolate to sequences longer than those seen in training. In this work, we explore State Space Models (SSMs) as a promising alternative by benchmarking two SSM-inspired architectures, Caduceus and Hawk, on long-range genomics modeling tasks under conditions parallel to a 50M parameter transformer baseline. We discover that SSMs match transformer performance and exhibit impressive zero-shot extrapolation across multiple tasks, handling contexts 10 to 100 times longer than those seen during training, indicating more generalizable representations better suited for modeling the long and complex human genome. Moreover, we demonstrate that these models can efficiently process sequences of 1M tokens on a single GPU, allowing for modeling entire genomic regions at once, even in labs with limited compute. Our findings establish SSMs as efficient and scalable for long-context genomic analysis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_06304 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Leveraging State Space Models in Long Range Genomics Popov, Matvei Kallala, Aymen Ramesh, Anirudha Hennouni, Narimane Khaitan, Shivesh Gentry, Rick Cohen, Alain-Sam Genomics Computer Vision and Pattern Recognition Machine Learning Long-range dependencies are critical for understanding genomic structure and function, yet most conventional methods struggle with them. Widely adopted transformer-based models, while excelling at short-context tasks, are limited by the attention module's quadratic computational complexity and inability to extrapolate to sequences longer than those seen in training. In this work, we explore State Space Models (SSMs) as a promising alternative by benchmarking two SSM-inspired architectures, Caduceus and Hawk, on long-range genomics modeling tasks under conditions parallel to a 50M parameter transformer baseline. We discover that SSMs match transformer performance and exhibit impressive zero-shot extrapolation across multiple tasks, handling contexts 10 to 100 times longer than those seen during training, indicating more generalizable representations better suited for modeling the long and complex human genome. Moreover, we demonstrate that these models can efficiently process sequences of 1M tokens on a single GPU, allowing for modeling entire genomic regions at once, even in labs with limited compute. Our findings establish SSMs as efficient and scalable for long-context genomic analysis. |
| title | Leveraging State Space Models in Long Range Genomics |
| topic | Genomics Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2504.06304 |