Effective Biological Representation Learning by Masking Gene Expression
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910272576290816 |
|---|---|
| author | Kenyon-Dean, Kian Selega, Alina Bendidi, Ihab Sorokin, Jordan M. Bertinetto, Luca Errington, David Donnella, Hayley Kraus, Oren |
| author_facet | Kenyon-Dean, Kian Selega, Alina Bendidi, Ihab Sorokin, Jordan M. Bertinetto, Luca Errington, David Donnella, Hayley Kraus, Oren |
| contents | RNA sequencing produces rich and diverse datasets of gene expression, offering compelling insights into cellular state and function that have many applications in drug discovery. Modeling such data is challenging due to inherent technical noise and experimental batch effects, as evidenced by many existing transcriptomic foundation models (FMs) underperforming relative to linear baselines. Such results raise the question of whether deep representation learning provides a distinct advantage over the direct use of raw transcript counts. Our work explores this by developing a new self-supervised model, TxFM, with a focus on inductive representation learning evaluations. TxFM employs a masked autoencoding approach tailored to diverse RNA-seq count data, and our ablation study empirically identifies crucial architecture configurations required for strong transfer performance. Additionally, we curate a public training corpus, DiverseRNA-1.4M, and find that TxFM trained on this curated dataset yields high-fidelity gene representations that outperform FMs trained on atlas-scale corpora over 100x larger. Overall, our results indicate that inductive self-supervised learning is a viable modeling approach for transcriptomics representation, provided a careful synthesis of model architecture and training data curation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_31562 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Effective Biological Representation Learning by Masking Gene Expression Kenyon-Dean, Kian Selega, Alina Bendidi, Ihab Sorokin, Jordan M. Bertinetto, Luca Errington, David Donnella, Hayley Kraus, Oren Machine Learning 68T07, 92D20 I.2.6; J.3 RNA sequencing produces rich and diverse datasets of gene expression, offering compelling insights into cellular state and function that have many applications in drug discovery. Modeling such data is challenging due to inherent technical noise and experimental batch effects, as evidenced by many existing transcriptomic foundation models (FMs) underperforming relative to linear baselines. Such results raise the question of whether deep representation learning provides a distinct advantage over the direct use of raw transcript counts. Our work explores this by developing a new self-supervised model, TxFM, with a focus on inductive representation learning evaluations. TxFM employs a masked autoencoding approach tailored to diverse RNA-seq count data, and our ablation study empirically identifies crucial architecture configurations required for strong transfer performance. Additionally, we curate a public training corpus, DiverseRNA-1.4M, and find that TxFM trained on this curated dataset yields high-fidelity gene representations that outperform FMs trained on atlas-scale corpora over 100x larger. Overall, our results indicate that inductive self-supervised learning is a viable modeling approach for transcriptomics representation, provided a careful synthesis of model architecture and training data curation. |
| title | Effective Biological Representation Learning by Masking Gene Expression |
| topic | Machine Learning 68T07, 92D20 I.2.6; J.3 |
| url | https://arxiv.org/abs/2605.31562 |