PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914112652443648 |
|---|---|
| author | Wu, Yan Wershof, Esther Schmon, Sebastian M Nassar, Marcel Osiński, Błażej Eksi, Ridvan Yan, Zichao Stark, Rory Zhang, Kun Graepel, Thore |
| author_facet | Wu, Yan Wershof, Esther Schmon, Sebastian M Nassar, Marcel Osiński, Błażej Eksi, Ridvan Yan, Zichao Stark, Rory Zhang, Kun Graepel, Thore |
| contents | We introduce a comprehensive framework for modeling single cell transcriptomic responses to perturbations, aimed at standardizing benchmarking in this rapidly evolving field. Our approach includes a modular and user-friendly model development and evaluation platform, a collection of diverse perturbational datasets, and a set of metrics designed to fairly compare models and dissect their performance. Through extensive evaluation of both published and baseline models across diverse datasets, we highlight the limitations of widely used models, such as mode collapse. We also demonstrate the importance of rank metrics which complement traditional model fit measures, such as RMSE, for validating model effectiveness. Notably, our results show that while no single model architecture clearly outperforms others, simpler architectures are generally competitive and scale well with larger datasets. Overall, this benchmarking exercise sets new standards for model evaluation, supports robust model development, and furthers the use of these models to simulate genetic and chemical screens for therapeutic discovery. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_10609 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis Wu, Yan Wershof, Esther Schmon, Sebastian M Nassar, Marcel Osiński, Błażej Eksi, Ridvan Yan, Zichao Stark, Rory Zhang, Kun Graepel, Thore Machine Learning Genomics We introduce a comprehensive framework for modeling single cell transcriptomic responses to perturbations, aimed at standardizing benchmarking in this rapidly evolving field. Our approach includes a modular and user-friendly model development and evaluation platform, a collection of diverse perturbational datasets, and a set of metrics designed to fairly compare models and dissect their performance. Through extensive evaluation of both published and baseline models across diverse datasets, we highlight the limitations of widely used models, such as mode collapse. We also demonstrate the importance of rank metrics which complement traditional model fit measures, such as RMSE, for validating model effectiveness. Notably, our results show that while no single model architecture clearly outperforms others, simpler architectures are generally competitive and scale well with larger datasets. Overall, this benchmarking exercise sets new standards for model evaluation, supports robust model development, and furthers the use of these models to simulate genetic and chemical screens for therapeutic discovery. |
| title | PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis |
| topic | Machine Learning Genomics |
| url | https://arxiv.org/abs/2408.10609 |