Benchmarking community drug response prediction models: datasets, models, tools, and metrics for cross-dataset generalization analysis
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913743744532480 |
|---|---|
| author | Partin, Alexander Vasanthakumari, Priyanka Narykov, Oleksandr Wilke, Andreas Koussa, Natasha Jones, Sara E. Zhu, Yitan Overbeek, Jamie C. Jain, Rajeev Fernando, Gayara Demini Sanchez-Villalobos, Cesar Garcia-Cardona, Cristina Mohd-Yusof, Jamaludin Chia, Nicholas Wozniak, Justin M. Ghosh, Souparno Pal, Ranadip Brettin, Thomas S. Weil, M. Ryan Stevens, Rick L. |
| author_facet | Partin, Alexander Vasanthakumari, Priyanka Narykov, Oleksandr Wilke, Andreas Koussa, Natasha Jones, Sara E. Zhu, Yitan Overbeek, Jamie C. Jain, Rajeev Fernando, Gayara Demini Sanchez-Villalobos, Cesar Garcia-Cardona, Cristina Mohd-Yusof, Jamaludin Chia, Nicholas Wozniak, Justin M. Ghosh, Souparno Pal, Ranadip Brettin, Thomas S. Weil, M. Ryan Stevens, Rick L. |
| contents | Deep learning (DL) and machine learning (ML) models have shown promise in drug response prediction (DRP), yet their ability to generalize across datasets remains an open question, raising concerns about their real-world applicability. Due to the lack of standardized benchmarking approaches, model evaluations and comparisons often rely on inconsistent datasets and evaluation criteria, making it difficult to assess true predictive capabilities. In this work, we introduce a benchmarking framework for evaluating cross-dataset prediction generalization in DRP models. Our framework incorporates five publicly available drug screening datasets, six standardized DRP models, and a scalable workflow for systematic evaluation. To assess model generalization, we introduce a set of evaluation metrics that quantify both absolute performance (e.g., predictive accuracy across datasets) and relative performance (e.g., performance drop compared to within-dataset results), enabling a more comprehensive assessment of model transferability. Our results reveal substantial performance drops when models are tested on unseen datasets, underscoring the importance of rigorous generalization assessments. While several models demonstrate relatively strong cross-dataset generalization, no single model consistently outperforms across all datasets. Furthermore, we identify CTRPv2 as the most effective source dataset for training, yielding higher generalization scores across target datasets. By sharing this standardized evaluation framework with the community, our study aims to establish a rigorous foundation for model comparison, and accelerate the development of robust DRP models for real-world applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_14356 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Benchmarking community drug response prediction models: datasets, models, tools, and metrics for cross-dataset generalization analysis Partin, Alexander Vasanthakumari, Priyanka Narykov, Oleksandr Wilke, Andreas Koussa, Natasha Jones, Sara E. Zhu, Yitan Overbeek, Jamie C. Jain, Rajeev Fernando, Gayara Demini Sanchez-Villalobos, Cesar Garcia-Cardona, Cristina Mohd-Yusof, Jamaludin Chia, Nicholas Wozniak, Justin M. Ghosh, Souparno Pal, Ranadip Brettin, Thomas S. Weil, M. Ryan Stevens, Rick L. Machine Learning Quantitative Methods Deep learning (DL) and machine learning (ML) models have shown promise in drug response prediction (DRP), yet their ability to generalize across datasets remains an open question, raising concerns about their real-world applicability. Due to the lack of standardized benchmarking approaches, model evaluations and comparisons often rely on inconsistent datasets and evaluation criteria, making it difficult to assess true predictive capabilities. In this work, we introduce a benchmarking framework for evaluating cross-dataset prediction generalization in DRP models. Our framework incorporates five publicly available drug screening datasets, six standardized DRP models, and a scalable workflow for systematic evaluation. To assess model generalization, we introduce a set of evaluation metrics that quantify both absolute performance (e.g., predictive accuracy across datasets) and relative performance (e.g., performance drop compared to within-dataset results), enabling a more comprehensive assessment of model transferability. Our results reveal substantial performance drops when models are tested on unseen datasets, underscoring the importance of rigorous generalization assessments. While several models demonstrate relatively strong cross-dataset generalization, no single model consistently outperforms across all datasets. Furthermore, we identify CTRPv2 as the most effective source dataset for training, yielding higher generalization scores across target datasets. By sharing this standardized evaluation framework with the community, our study aims to establish a rigorous foundation for model comparison, and accelerate the development of robust DRP models for real-world applications. |
| title | Benchmarking community drug response prediction models: datasets, models, tools, and metrics for cross-dataset generalization analysis |
| topic | Machine Learning Quantitative Methods |
| url | https://arxiv.org/abs/2503.14356 |