Beyond Conditional Computation: Retrieval-Augmented Genomic Foundation Models with Gengram
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917233211473920 |
|---|---|
| author | Xu, Huinan Feng, Xuyang Chen, Junhong Liu, Junchen Deng, Kaiwen Ding, Kai Long, Shengning Shuai, Jiaxue Li, Zhaorong Liu, Shiping Xue, Guirong Xiao, Zhan |
| author_facet | Xu, Huinan Feng, Xuyang Chen, Junhong Liu, Junchen Deng, Kaiwen Ding, Kai Long, Shengning Shuai, Jiaxue Li, Zhaorong Liu, Shiping Xue, Guirong Xiao, Zhan |
| contents | Current genomic foundation models (GFMs) rely on extensive neural computation to implicitly approximate conserved biological motifs from single-nucleotide inputs. We propose Gengram, a conditional memory module that introduces an explicit and highly efficient lookup primitive for multi-base motifs via a genomic-specific hashing scheme, establishing genomic "syntax". Integrated into the backbone of state-of-the-art GFMs, Gengram achieves substantial gains (up to 14%) across several functional genomics tasks. The module demonstrates robust architectural generalization, while further inspection of Gengram's latent space reveals the emergence of meaningful representations that align closely with fundamental biological knowledge. By establishing structured motif memory as a modeling primitive, Gengram simultaneously boosts empirical performance and mechanistic interpretability, providing a scalable and biology-aligned pathway for the next generation of GFMs. The code is available at https://github.com/zhejianglab/Genos, and the model checkpoint is available at https://huggingface.co/ZhejiangLab/Gengram. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_22203 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Beyond Conditional Computation: Retrieval-Augmented Genomic Foundation Models with Gengram Xu, Huinan Feng, Xuyang Chen, Junhong Liu, Junchen Deng, Kaiwen Ding, Kai Long, Shengning Shuai, Jiaxue Li, Zhaorong Liu, Shiping Xue, Guirong Xiao, Zhan Genomics Artificial Intelligence Current genomic foundation models (GFMs) rely on extensive neural computation to implicitly approximate conserved biological motifs from single-nucleotide inputs. We propose Gengram, a conditional memory module that introduces an explicit and highly efficient lookup primitive for multi-base motifs via a genomic-specific hashing scheme, establishing genomic "syntax". Integrated into the backbone of state-of-the-art GFMs, Gengram achieves substantial gains (up to 14%) across several functional genomics tasks. The module demonstrates robust architectural generalization, while further inspection of Gengram's latent space reveals the emergence of meaningful representations that align closely with fundamental biological knowledge. By establishing structured motif memory as a modeling primitive, Gengram simultaneously boosts empirical performance and mechanistic interpretability, providing a scalable and biology-aligned pathway for the next generation of GFMs. The code is available at https://github.com/zhejianglab/Genos, and the model checkpoint is available at https://huggingface.co/ZhejiangLab/Gengram. |
| title | Beyond Conditional Computation: Retrieval-Augmented Genomic Foundation Models with Gengram |
| topic | Genomics Artificial Intelligence |
| url | https://arxiv.org/abs/2601.22203 |