PyTond: Efficient Python Data Science on the Shoulders of Databases
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917723325333504 |
|---|---|
| author | Shahrokhi, Hesam Kaboli, Amirali Ghorbani, Mahdi Shaikhha, Amir |
| author_facet | Shahrokhi, Hesam Kaboli, Amirali Ghorbani, Mahdi Shaikhha, Amir |
| contents | Python data science libraries such as Pandas and NumPy have recently gained immense popularity. Although these libraries are feature-rich and easy to use, their scalability limitations require more robust computational resources. In this paper, we present PyTond, an efficient approach to push the processing of data science workloads down into the database engines that are already known for their big data handling capabilities. Compared to the previous work, by introducing TondIR, our approach can capture a more comprehensive set of workloads and data layouts. Moreover, by doing IR-level optimizations, we generate better SQL code that improves the query processing by the underlying database engine. Our evaluation results show promising performance improvement compared to Python and other alternatives for diverse data science workloads. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_11616 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | PyTond: Efficient Python Data Science on the Shoulders of Databases Shahrokhi, Hesam Kaboli, Amirali Ghorbani, Mahdi Shaikhha, Amir Databases Programming Languages Python data science libraries such as Pandas and NumPy have recently gained immense popularity. Although these libraries are feature-rich and easy to use, their scalability limitations require more robust computational resources. In this paper, we present PyTond, an efficient approach to push the processing of data science workloads down into the database engines that are already known for their big data handling capabilities. Compared to the previous work, by introducing TondIR, our approach can capture a more comprehensive set of workloads and data layouts. Moreover, by doing IR-level optimizations, we generate better SQL code that improves the query processing by the underlying database engine. Our evaluation results show promising performance improvement compared to Python and other alternatives for diverse data science workloads. |
| title | PyTond: Efficient Python Data Science on the Shoulders of Databases |
| topic | Databases Programming Languages |
| url | https://arxiv.org/abs/2407.11616 |