NApy: Efficient Statistics in Python for Large-Scale Heterogeneous Data with Enhanced Support for Missing Data
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909598249648128 |
|---|---|
| author | Woller, Fabian Arend, Lis Fuchsberger, Christian List, Markus Blumenthal, David B. |
| author_facet | Woller, Fabian Arend, Lis Fuchsberger, Christian List, Markus Blumenthal, David B. |
| contents | Existing Python libraries and tools lack the ability to efficiently compute statistical test results for large datasets in the presence of missing values. This presents an issue as soon as constraints on runtime and memory availability become essential considerations for a particular usecase. Relevant research areas where such limitations arise include interactive tools and databases for exploratory analysis of biomedical data. To address this problem, we present the Python package NApy, which relies on a Numba and C++ backend with OpenMP parallelization to enable scalable statistical testing for mixed-type datasets in the presence of missing values. Both with respect to runtime and memory consumption, NApy outperforms competitor tools and baseline implementations with naive Python-based parallelization by orders of magnitude, thereby enabling on-the-fly analyses in interactive applications. NApy is publicly available at https://github.com/DyHealthNet/NApy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_00448 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | NApy: Efficient Statistics in Python for Large-Scale Heterogeneous Data with Enhanced Support for Missing Data Woller, Fabian Arend, Lis Fuchsberger, Christian List, Markus Blumenthal, David B. Mathematical Software Distributed, Parallel, and Cluster Computing Performance Existing Python libraries and tools lack the ability to efficiently compute statistical test results for large datasets in the presence of missing values. This presents an issue as soon as constraints on runtime and memory availability become essential considerations for a particular usecase. Relevant research areas where such limitations arise include interactive tools and databases for exploratory analysis of biomedical data. To address this problem, we present the Python package NApy, which relies on a Numba and C++ backend with OpenMP parallelization to enable scalable statistical testing for mixed-type datasets in the presence of missing values. Both with respect to runtime and memory consumption, NApy outperforms competitor tools and baseline implementations with naive Python-based parallelization by orders of magnitude, thereby enabling on-the-fly analyses in interactive applications. NApy is publicly available at https://github.com/DyHealthNet/NApy. |
| title | NApy: Efficient Statistics in Python for Large-Scale Heterogeneous Data with Enhanced Support for Missing Data |
| topic | Mathematical Software Distributed, Parallel, and Cluster Computing Performance |
| url | https://arxiv.org/abs/2505.00448 |