Scalable, reproducible, and cost-effective processing of large-scale medical imaging datasets
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916370622447616 |
|---|---|
| author | Kim, Michael E. Ramadass, Karthik Gao, Chenyu Kanakaraj, Praitayini Newlin, Nancy R. Rudravaram, Gaurav Schilling, Kurt G. Dewey, Blake E. Archer, Derek Hohman, Timothy J. Li, Zhiyuan Bao, Shunxing Landman, Bennett A. Khairi, Nazirah Mohd |
| author_facet | Kim, Michael E. Ramadass, Karthik Gao, Chenyu Kanakaraj, Praitayini Newlin, Nancy R. Rudravaram, Gaurav Schilling, Kurt G. Dewey, Blake E. Archer, Derek Hohman, Timothy J. Li, Zhiyuan Bao, Shunxing Landman, Bennett A. Khairi, Nazirah Mohd |
| contents | Curating, processing, and combining large-scale medical imaging datasets from national studies is a non-trivial task due to the intense computation and data throughput required, variability of acquired data, and associated financial overhead. Existing platforms or tools for large-scale data curation, processing, and storage have difficulty achieving a viable cost-to-scale ratio of computation speed for research purposes, either being too slow or too expensive. Additionally, management and consistency of processing large data in a team-driven manner is a non-trivial task. We design a BIDS-compliant method for an efficient and robust data processing pipeline of large-scale diffusion-weighted and T1-weighted MRI data compatible with low-cost, high-efficiency computing systems. Our method accomplishes automated querying of data available for processing and process running in a consistent and reproducible manner that has long-term stability, while using heterogenous low-cost computational resources and storage systems for efficient processing and data transfer. We demonstrate how our organizational structure permits efficiency in a semi-automated data processing pipeline and show how our method is comparable in processing time to cloud-based computation while being almost 20 times more cost-effective. Our design allows for fast data throughput speeds and low latency to reduce the time for data transfer between storage servers and computation servers, achieving an average of 0.60 Gb/s compared to 0.33 Gb/s for using cloud-based processing methods. The design of our workflow engine permits quick process running while maintaining flexibility to adapt to newly acquired data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_14611 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Scalable, reproducible, and cost-effective processing of large-scale medical imaging datasets Kim, Michael E. Ramadass, Karthik Gao, Chenyu Kanakaraj, Praitayini Newlin, Nancy R. Rudravaram, Gaurav Schilling, Kurt G. Dewey, Blake E. Archer, Derek Hohman, Timothy J. Li, Zhiyuan Bao, Shunxing Landman, Bennett A. Khairi, Nazirah Mohd Distributed, Parallel, and Cluster Computing Databases Curating, processing, and combining large-scale medical imaging datasets from national studies is a non-trivial task due to the intense computation and data throughput required, variability of acquired data, and associated financial overhead. Existing platforms or tools for large-scale data curation, processing, and storage have difficulty achieving a viable cost-to-scale ratio of computation speed for research purposes, either being too slow or too expensive. Additionally, management and consistency of processing large data in a team-driven manner is a non-trivial task. We design a BIDS-compliant method for an efficient and robust data processing pipeline of large-scale diffusion-weighted and T1-weighted MRI data compatible with low-cost, high-efficiency computing systems. Our method accomplishes automated querying of data available for processing and process running in a consistent and reproducible manner that has long-term stability, while using heterogenous low-cost computational resources and storage systems for efficient processing and data transfer. We demonstrate how our organizational structure permits efficiency in a semi-automated data processing pipeline and show how our method is comparable in processing time to cloud-based computation while being almost 20 times more cost-effective. Our design allows for fast data throughput speeds and low latency to reduce the time for data transfer between storage servers and computation servers, achieving an average of 0.60 Gb/s compared to 0.33 Gb/s for using cloud-based processing methods. The design of our workflow engine permits quick process running while maintaining flexibility to adapt to newly acquired data. |
| title | Scalable, reproducible, and cost-effective processing of large-scale medical imaging datasets |
| topic | Distributed, Parallel, and Cluster Computing Databases |
| url | https://arxiv.org/abs/2408.14611 |