BMFM-RNA: whole-cell expression decoding improves transcriptomic foundation models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914423523770368 |
|---|---|
| author | Danziger, Michael M. Dandala, Bharath Gurev, Viatcheslav Madgwick, Matthew Ravid, Sivan Rumbell, Tim Koseki, Akira Kozlovski, Tal Tsou, Ching-Huei Barkan, Ella Biswas, Tanwi Xu, Jielin Shimoni, Yishai Hu, Jianying Rosen-Zvi, Michal |
| author_facet | Danziger, Michael M. Dandala, Bharath Gurev, Viatcheslav Madgwick, Matthew Ravid, Sivan Rumbell, Tim Koseki, Akira Kozlovski, Tal Tsou, Ching-Huei Barkan, Ella Biswas, Tanwi Xu, Jielin Shimoni, Yishai Hu, Jianying Rosen-Zvi, Michal |
| contents | Transcriptomic foundation models pretrained with masked language modeling can achieve low pretraining loss yet produce poor cell representations for downstream tasks. We introduce whole-cell expression decoding (WCED), where models reconstruct the entire gene vocabulary from a single CLS token embedding, even with limited inputs, creating a maximally informative bottleneck. WCED consistently outperforms MLM on all downstream metrics despite higher reconstruction error during training. Gene-level error tracking reveals that both methods preferentially learn genes whose expression co-varies with stable transcriptional programs rather than those driven by transient factors. We further add hierarchical cross-entropy loss that exploits Cell Ontology structure for zero-shot annotation at multiple granularity levels. Models trained with these objectives achieve best overall performance across CZI benchmarks, on zero-shot batch integration and linear probing cell-type annotation. Methods are implemented in biomed-multi-omic ( https://github.com/BiomedSciAI/biomed-multi-omic ), an open-source framework for transcriptomic foundation model development. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_14861 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | BMFM-RNA: whole-cell expression decoding improves transcriptomic foundation models Danziger, Michael M. Dandala, Bharath Gurev, Viatcheslav Madgwick, Matthew Ravid, Sivan Rumbell, Tim Koseki, Akira Kozlovski, Tal Tsou, Ching-Huei Barkan, Ella Biswas, Tanwi Xu, Jielin Shimoni, Yishai Hu, Jianying Rosen-Zvi, Michal Genomics Artificial Intelligence Quantitative Methods Transcriptomic foundation models pretrained with masked language modeling can achieve low pretraining loss yet produce poor cell representations for downstream tasks. We introduce whole-cell expression decoding (WCED), where models reconstruct the entire gene vocabulary from a single CLS token embedding, even with limited inputs, creating a maximally informative bottleneck. WCED consistently outperforms MLM on all downstream metrics despite higher reconstruction error during training. Gene-level error tracking reveals that both methods preferentially learn genes whose expression co-varies with stable transcriptional programs rather than those driven by transient factors. We further add hierarchical cross-entropy loss that exploits Cell Ontology structure for zero-shot annotation at multiple granularity levels. Models trained with these objectives achieve best overall performance across CZI benchmarks, on zero-shot batch integration and linear probing cell-type annotation. Methods are implemented in biomed-multi-omic ( https://github.com/BiomedSciAI/biomed-multi-omic ), an open-source framework for transcriptomic foundation model development. |
| title | BMFM-RNA: whole-cell expression decoding improves transcriptomic foundation models |
| topic | Genomics Artificial Intelligence Quantitative Methods |
| url | https://arxiv.org/abs/2506.14861 |