scDataset: Scalable Data Loading for Deep Learning on Large-Scale Single-Cell Omics
Fuente:
arXiv
Salvato in:
| Autori principali: | , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914290519244800 |
|---|---|
| author | D'Ascenzo, Davide di Montesano, Sebastiano Cultrera |
| author_facet | D'Ascenzo, Davide di Montesano, Sebastiano Cultrera |
| contents | Training deep learning models on single-cell datasets with hundreds of millions of cells requires loading data from disk, as these datasets exceed available memory. While random sampling provides the data diversity needed for effective training, it is prohibitively slow due to the random access pattern overhead, whereas sequential streaming achieves high throughput but introduces biases that degrade model performance. We present scDataset, a PyTorch data loader that enables efficient training from on-disk data with seamless integration across diverse storage formats. Our approach combines block sampling and batched fetching to achieve quasi-random sampling that balances I/O efficiency with minibatch diversity. On Tahoe-100M, a dataset of 100 million cells, scDataset achieves more than two orders of magnitude speedup compared to true random sampling while working directly with AnnData files. We provide theoretical bounds on minibatch diversity and empirically show that scDataset matches the performance of true random sampling across multiple classification tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_01883 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | scDataset: Scalable Data Loading for Deep Learning on Large-Scale Single-Cell Omics D'Ascenzo, Davide di Montesano, Sebastiano Cultrera Machine Learning Artificial Intelligence Databases Genomics Quantitative Methods 68T07, 68P05, 92C40 I.2.6; H.2.8; J.3 Training deep learning models on single-cell datasets with hundreds of millions of cells requires loading data from disk, as these datasets exceed available memory. While random sampling provides the data diversity needed for effective training, it is prohibitively slow due to the random access pattern overhead, whereas sequential streaming achieves high throughput but introduces biases that degrade model performance. We present scDataset, a PyTorch data loader that enables efficient training from on-disk data with seamless integration across diverse storage formats. Our approach combines block sampling and batched fetching to achieve quasi-random sampling that balances I/O efficiency with minibatch diversity. On Tahoe-100M, a dataset of 100 million cells, scDataset achieves more than two orders of magnitude speedup compared to true random sampling while working directly with AnnData files. We provide theoretical bounds on minibatch diversity and empirically show that scDataset matches the performance of true random sampling across multiple classification tasks. |
| title | scDataset: Scalable Data Loading for Deep Learning on Large-Scale Single-Cell Omics |
| topic | Machine Learning Artificial Intelligence Databases Genomics Quantitative Methods 68T07, 68P05, 92C40 I.2.6; H.2.8; J.3 |
| url | https://arxiv.org/abs/2506.01883 |